Open-Weight LLM · Private & Custom AI
Step-3.5-Flash
Sparse MoE reasoning engine for private deployment—11B active parameters, 256K context, purpose-built for agentic workflows and long-horizon operational tasks.
Step 3.5 Flash is a 196B MoE foundation model activating only 11B parameters per token, designed for high-throughput reasoning, coding, and agentic work. For ops teams, it offers frontier-grade intelligence (competitive with proprietary models on SWE-bench, Terminal-Bench, and reasoning benchmarks) at a hardware footprint suitable for on-premises or private cloud deployment. The sparse architecture trades minimal latency loss for massive parameter efficiency—critical for companies needing custom AI systems without cloud vendor lock-in.
Model facts
Private deployment
Run Step-3.5-Flash in your own environment
Self-hosting Step 3.5 Flash requires approximately 400–550 GB VRAM (FP16/BF16) or 200–275 GB (INT8) per inference node on a Hopper/Ada GPU cluster, or high-end consumer multi-GPU setups (e.g., Mac Studio M4 Max for smaller batch sizes). Apache 2.0 license permits private deployment without commercial restrictions. The architecture is optimized for low-batch, high-throughput inference; expect 100–350 tok/s depending on context length and hardware. A company hosting this privately gains full data residency (prompts, outputs, fine-tuning data remain in their environment), eliminates per-token API costs at scale, and retains ability to customize, quantize, or fine-tune the model—critical for regulated industries or proprietary reasoning tasks.
Operational AI use cases
Code Review & DevOps Automation
Deploying Step 3.5 Flash as a private code-review and CI/CD reasoning agent. The model's 74.4% SWE-bench Verified score and native support for 256K context means it can analyze entire codebases, PR diffs, and test suites in a single pass. Ops teams use it to flag merge-blocking issues, auto-generate runbooks for deployment failures, and suggest configuration fixes without sending code to external APIs—critical for financial services, healthcare, or defense contractors.
Multi-Step Operational Triage & Ticket Routing
Implement a private AI triage agent that ingests support tickets, runbooks, and internal knowledge bases (fed into 256K context), then reasons through root cause and routes to the right team. Step 3.5 Flash's 88.2 τ²-Bench score (agent reasoning) and Multi-Token Prediction deliver sub-second latency triage decisions. Reduces manual ticket assignment, improves MTTR, keeps customer data off third-party servers.
Financial & Compliance Document Processing
Use the model to parse long contracts, audit logs, regulatory filings, and financial statements in-house. 256K context allows ingestion of 10,000+ page documents in one shot. The model reason over clauses, flag compliance mismatches, and extract key terms—all without exposing sensitive data to cloud vendors. Sparse architecture keeps inference cost low for batch processing of high-volume doc workflows.
Custom AI
As a base for custom AI
Step 3.5 Flash is a strong foundation for proprietary AI products requiring agentic reasoning and code understanding. Fine-tune it on domain-specific datasets (financial analysis, security testing, technical support) while keeping all training data private. Its sparse MoE design means parameter-efficient adaptation—add custom experts or LoRA layers without retraining the full 196B model. Companies building white-label AI products can deploy it as a managed service, or embed it in desktop/mobile applications. The Apache 2.0 license permits commercial products and redistribution, provided derivative licenses are clear.
In the operating system
Where it fits
In an LLM.co-style AI operating system, Step 3.5 Flash sits at the **reasoning & agentic layer**—handling multi-step workflows, long-context document analysis, and code-level reasoning that simpler chat or embedding models cannot. It feeds into orchestration layers (agent schedulers, memory systems) and integrates with operational data stores (knowledge bases, ticket systems, logs). Its 256K context and 11B active parameters make it the backbone for **knowledge agents** (internal docs + reasoning) and **workflow automators** (multi-hop ops tasks). For comparison: use smaller models (7B-13B dense) for classification/routing; use Step 3.5 Flash when depth and context are non-negotiable.
Data control & security
Self-hosting Step 3.5 Flash ensures data never transits external APIs—all prompts, reasoning chains, and outputs stay within your VPC or on-premises infrastructure. This is an **architectural advantage**, not a security guarantee from the model itself. You remain responsible for network segmentation, access controls, and audit logging. For regulated workloads (HIPAA, PCI, FedRAMP), private deployment eliminates third-party data-processing agreements and audit dependencies. The model has no built-in encryption or differential privacy; those are deployment-layer choices. No claims of inherent robustness to jailbreaks or adversarial prompts—apply standard red-team practices. Requires review of your deployment security posture before handling sensitive data.
Hardware footprint
**Estimate (FP16/BF16)**: ~450 GB VRAM per inference node (spans 4× H100 or 2× H200). **INT8 quantization**: ~225 GB. **Streaming/lower batch**: can run on 2× H100 (320 GB combined) or high-end multi-GPU consumer setups (4× RTX 6000 Ada). **CPU inference**: technically possible (e.g., llama.cpp + MoE support) but impractical—expect 1–5 tok/s. **Activation memory**: actual per-token usage is ~11B params (~22 GB FP16), but MoE routing and key-value cache for 256K context require buffer space. Throughput scales with batch size; single-token latency ~50–100 ms on a single H100 at 256K context (hybrid attention). Multi-Token Prediction (MTP-3) reduces wall-clock latency by predicting 4 tokens per forward pass.
Integration
Step 3.5 Flash integrates via standard OpenAI-compatible APIs (used by OpenRouter and StepFun's platform). For private deployment, wrap it with vLLM, TGI, or LM Studio for fast batch/streaming inference. Connect via LangChain, LlamaIndex, or custom Python clients to orchestration frameworks (n8n, Temporal, Airflow). For ops automation: hook it into Jira/Linear (via webhooks) for ticket triage, into GitHub/GitLab for code review agents, into Slack for real-time decision support. Supports LoRA adapters and quantization (INT8, NF4) for fine-tuning and resource optimization. No native function calling; use prompt engineering or wrapper libraries (LangChain tools) to bind it to external APIs. Context length of 256K means batch large datasets but manage token-per-second throughput on smaller GPU clusters.
When it's not the right fit
- —You need sub-50ms latency on single-token requests and cannot tolerate high-batch aggregation. Sparse MoE routing adds compute overhead; dense 11B-13B models (Phi-4, Mistral Nemo) are faster for real-time chat.
- —Your team lacks GPU infrastructure or ops expertise for private LLM deployment. Hosted APIs (OpenRouter, StepFun platform) are simpler but introduce data residency and cost-per-token constraints.
- —You require certified compliance (SOC 2, HIPAA) or want vendor indemnification. Self-hosted LLMs shift compliance burden to your team; no built-in auditing or regulatory hooks.
- —Your task is lightweight classification, embedding, or summarization (<10K context). Smaller, faster models (Phi-3, Jina Embeddings) are cheaper and lower operational overhead.
Alternatives to consider
DeepSeek-V3 (671B MoE, 37B active)
Stronger long-context reasoning and larger active parameter set (37B vs. 11B), but 6× higher decoding cost. Choose if reasoning depth outweighs latency/cost. Slightly lower code performance (SWE-bench 73.1 vs. 74.4).
Llama 3.1 (405B dense, 70B/8B variants)
Fully open, MIT license, no MoE complexity. Better for teams without MoE inference optimization. Dense 70B variant is slower but simpler to deploy; less suitable for massive context windows or low-batch-size efficiency.
Mixtral 8x22B (141B MoE, ~39B active)
Established MoE architecture, well-optimized in vLLM. Smaller total parameter footprint (~141B vs. 196B), suitable for tighter GPU budgets. Trade-off: less sophisticated reasoning and shorter effective context (200K advertised, less stable at max).
Related open models
FAQ
Can we run Step 3.5 Flash entirely on-premises, or must we use OpenRouter/StepFun's hosted API?
You can fully self-host. Apache 2.0 license permits private deployment. Requires GPU cluster (~450 GB VRAM for FP16 on Hopper), integration of vLLM or TGI, and operational overhead for scaling/failover. Use hosted APIs if you prefer vendor management; choose private deployment if data residency or customization (fine-tuning, LoRA) is mandatory.
Is Step 3.5 Flash suitable for commercial AI products?
Yes. Apache 2.0 is permissive for commercial use, including proprietary products and SaaS. You can fine-tune, quantize, and redistribute (must maintain license attribution). No royalties or usage restrictions. Verify your supply chain: OpenRouter and StepFun offerings allow commercial deployment; redistribute derivatives with clear licensing.
How does MoE routing affect inference latency and cost?
Routing adds ~5–10% overhead per forward pass, but sparse activation (11B of 196B) reduces total compute by ~85% vs. a dense 196B model. Net result: ~100–350 tok/s on modern GPUs (competitive with dense 30–40B models but at lower VRAM). Cost per token is lower if you self-host (no per-token API pricing); hosted APIs charge at their standard rate (check OpenRouter/StepFun pricing).
What if our ops team has no LLM deployment experience?
Use OpenRouter's free trial or StepFun's hosted API first—zero infrastructure setup. Once your use case is validated, partner with an LLM ops vendor (vLLM consultants, Anyscale, Lambda Labs) to build a private instance. LLM.co can guide architecture design for your ops automation layer.
Ready to Build a Private AI Operating System?
Step 3.5 Flash is built for custom ops AI and private deployment. LLM.co helps you architect reasoning agents, integrate with internal systems, and keep data in your environment. Let's design your AI stack.