Open-Weight LLM · Private & Custom AI
Qwen3-235B-A22B-FP8
A 235B mixture-of-experts model with native thinking/non-thinking modes—designed for companies building private reasoning agents, complex workflow automation, and multilingual ops AI without external API dependencies.
Qwen3-235B-A22B is Alibaba's latest dense-MoE LLM (22B activated parameters) supporting dynamic reasoning modes, 100+ languages, and native agent tool-integration. For ops teams, it's a private-deployable base for automating high-stakes decisions (compliance, technical support, content triage) while keeping all reasoning and data in-house.
Model facts
Private deployment
Run Qwen3-235B-A22B-FP8 in your own environment
Self-hosting requires ~47–94 GB VRAM (FP8 quantization reduces overhead; full bfloat16 is memory-intensive). Inference frameworks SGLang (≥0.4.6.post1), vLLM (≥0.8.5), and transformers (≥4.51.0) all support it; local wrappers (Ollama, LMStudio, llama.cpp, KTransformers) also available. A company runs the full model on internal GPU clusters—reasoning happens inside the firewall, no tokens leave the environment. Key trade-off: FP8 quantization introduces fine-grained precision loss (block size 128) and has known issues with distributed inference in transformers (may require `CUDA_LAUNCH_BLOCKING=1`). Advantage: you control versioning, can audit/customize prompts, and comply with data residency rules.
Operational AI use cases
Compliance & Risk Triage Agent
Enable finance/legal teams to auto-route regulatory filings, contracts, and incident reports. Thinking mode engages for complex logical inference (e.g., "Is this clause a breach of X policy?"); non-thinking mode handles rapid fact-retrieval. Agents call internal compliance databases and decision trees. All reasoning stays private; audit trails remain in your system.
Technical Support Escalation Workflow
Route customer support tickets by severity and domain. Thinking mode diagnoses root cause (debugging logs, error patterns); non-thinking mode answers FAQs in real-time. Agents integrate with your ticketing system (Zendesk, Jira), knowledge base, and asset inventory. Multilingual support handles global support ops; no external inference = lower latency and full data control.
Content & Document Processing Pipeline
Ingest internal docs (SOPs, policies, technical specs) and auto-generate summaries, FAQs, training materials, or compliance checklists. Thinking mode ensures logical consistency in long-form outputs; non-thinking mode batches simple transformations. Integrate with document management systems (SharePoint, Confluence). Data never leaves your environment; ideal for regulated industries (healthcare, finance, gov).
Custom AI
As a base for custom AI
Strong base for custom ops AI products. Its mixture-of-experts design and thinking/non-thinking modes allow product teams to fine-tune it on domain-specific data (e.g., internal SLAs, company terminology, regulatory frameworks) without retraining from scratch. Developers can build tool-calling wrappers, prompt templates, and agent orchestration on top. The model's support for long context (up to 131k tokens with YaRN) and 100+ languages makes it adaptable for multi-region, multi-lingual custom AI. Quantization path (FP8 provided) enables cost-efficient deployment at scale.
In the operating system
Where it fits
In an AI operating system (AOS), Qwen3-235B acts as the **reasoning + language foundation** at the core of the agent and workflow layers. It powers tool-calling agents (thinking mode for complex logic, non-thinking for speed), orchestrates multi-step operational processes, and serves as the base for fine-tuned domain models. Data flows from your knowledge layer (docs, databases) → Qwen3 (private inference) → workflow execution layer (APIs, systems, approvals). Its thinking/non-thinking toggle lets the AOS route requests intelligently: high-stakes decisions engage thinking; routine ops use non-thinking for throughput.
Data control & security
Self-hosting on your infrastructure means no tokens, prompts, or outputs pass through third-party cloud services. Sensitive operational data (customer records, financial data, proprietary processes) stays inside your firewall. However: security posture depends on your deployment—GPU cluster access control, network isolation, and secrets management are your responsibility. FP8 quantization is a trade-off: it reduces VRAM and cost but introduces floating-point precision loss (block size 128) that may subtly affect reasoning quality in edge cases. No built-in encryption or compliance certifications from the model itself; you must layer those via infrastructure (TLS, database encryption, audit logging).
Hardware footprint
**Estimate (verify on your hardware).** FP8 quantization (~235B params, 128-bit block quantization): ~47–60 GB VRAM for inference (single GPU or multi-GPU). Full bfloat16 (unquantized): ~94–120 GB VRAM. Activated parameters per token: ~22B (MoE routing). Throughput depends on hardware (A100 40GB, H100 80GB, or clusters); expect lower than dense models due to routing overhead, faster than loading 235B dense. Batch inference (vLLM paged attention) can improve GPU utilization.
Integration
Qwen3-235B integrates via standard transformers/vLLM/SGLang APIs. For ops: (1) Wrap inference in an OpenAI-compatible endpoint (vLLM/SGLang do this natively) so your ops stack (n8n, Zapier, custom Python agents) connects via standard REST/SDK. (2) Tokenizer is standard; Hugging Face transformers ≥4.51.0 required. (3) Tool-calling via structured prompts (JSON schemas in system messages or via agent frameworks like LangChain, LlamaIndex). (4) For thinking mode, set Temperature=0.6, TopP=0.95, TopK=20; for non-thinking, Temperature=0.7, TopP=0.8 (greedy decoding breaks thinking mode). (5) Long-context use requires careful prompt engineering; YaRN extension to 131k tokens works but may introduce latency. (6) FP8 quantization: known issues with multi-GPU distributed inference in transformers; test on your hardware before production.
When it's not the right fit
- —Ultra-low latency required: MoE routing and (especially) thinking mode add latency vs. smaller dense models. For <100ms SLA, consider Qwen2.5-7B or similar.
- —Real-time streaming responses are the only goal: thinking mode blocks response output until reasoning is done; non-thinking is better but still slower than highly optimized smaller models.
- —Extreme resource constraints: 47+ GB VRAM is not 'edge' or embedded. If you need sub-10GB deployment, use quantized smaller models (Qwen2.5-3B, Phi).
- —Prefers vendor lock-in or managed inference: private deployment means you own ops (monitoring, scaling, patching). If you want serverless, consider cloud API services instead.
Alternatives to consider
Meta Llama 3.1 405B or 70B
Larger dense option (405B) or smaller dense (70B). No native thinking mode, but strong reasoning. Fully permissive license (Llama 2 Community License or newer). Simpler to deploy (no MoE routing complexity), but larger models demand more VRAM. Better if you don't need private reasoning toggle.
DeepSeek-R1-Distill (70B/32B quantized)
Also offers thinking/reasoning mode (similar to Qwen3's `<think>` block). Smaller than Qwen3-235B, easier to fit on single GPU. Apache 2.0 license. Good alternative if you want reasoning but smaller footprint; trade-off is less dense knowledge and fewer languages.
Mistral Large 123B or Mixtral 8x22B
Dense (123B) or MoE (8x22B) options from Mistral. Strong instruction-following and lower latency than Qwen3-235B. Mistral License (permissive for commercial use). No thinking mode, but faster inference if that's your bottleneck. Simpler ops, but less reasoning capability.
Related open models
FAQ
Can we run Qwen3-235B-FP8 entirely on-premise, and keep all data private?
Yes. Deploy it on your GPU cluster using vLLM, SGLang, or transformers. All inference, tokens, and reasoning happen inside your firewall. You own the data pipeline, logging, and audit trail. FP8 quantization reduces VRAM (~47–60 GB) vs. full precision. Key caveat: you're responsible for securing the inference cluster, managing updates, and monitoring performance.
Can we use Qwen3-235B-FP8 in a commercial product we build?
Yes. Apache 2.0 license permits commercial use, modification, and distribution (with attribution and license disclosure). You can fine-tune it on proprietary data, wrap it in a commercial product, and sell that product—no royalties or restrictions. Just include the Apache 2.0 notice in your product. Gating is disabled (model is public), so no approval process.
What's the difference between thinking and non-thinking mode operationally?
Thinking mode (`enable_thinking=True`) generates internal reasoning (`<think>...</think>`) before the final response—slower, more reliable for complex logic (math, debugging, compliance decisions). Non-thinking mode (`enable_thinking=False`) skips reasoning and answers immediately—faster, suitable for fact retrieval or routine ops. Toggle per request via `enable_thinking` parameter or soft-switch via `/think` / `/no_think` in prompts.
What are the known issues with FP8 quantization in production?
Fine-grained FP8 (block size 128) reduces precision uniformly. In transformers, multi-GPU distributed inference can fail; use `CUDA_LAUNCH_BLOCKING=1` to work around. vLLM and SGLang handle it better. Test FP8 thoroughly on your target hardware before deploying to production; some edge-case reasoning tasks may degrade slightly due to quantization loss.
Build Private Ops AI with Qwen3-235B
Ready to deploy a reasoning LLM entirely in your environment? LLM.co helps you integrate Qwen3-235B into custom workflows, automate operational decisions, and keep sensitive data private. Let's architect your AI operating system—no vendor lock-in, full control.