Open-Weight LLM · Private & Custom AI
Qwen3-235B-A22B-Instruct-2507-AWQ
A 235B MoE foundation model optimized for private deployment at scale, delivering competitive performance on reasoning, coding, and multi-language tasks while keeping inference under customer control.
Qwen3-235B-A22B-Instruct-2507-AWQ is a 4-bit quantized Mixture-of-Experts model (235B total, 22B activated) with native 262K context support and strong benchmarks across reasoning, coding, and knowledge tasks. For ops teams, it's a viable private alternative to closed APIs—deployable on 8-GPU clusters, Apache-2.0 licensed, and designed for vLLM inference at enterprise scale.
Model facts
Private deployment
Run Qwen3-235B-A22B-Instruct-2507-AWQ in your own environment
Self-host via vLLM (≥0.9.2) on 8× H100/A100 GPUs in a single node. Model card provides exact launch config with tensor parallelism, expert parallelism, and 262K context. No phone-home dependencies. Data stays in your VPC; inference logs and outputs remain fully internal. Requires ~116GB disk (quantized), ~200–250GB VRAM peak (8 GPUs, estimate). Ideal for companies wanting LLM inference without external API calls.
Operational AI use cases
Internal Knowledge Agent & Document Automation
Use as backbone for a private retrieval-augmented agent over internal docs (policies, runbooks, contracts). Qwen3's 262K context handles multi-document reasoning; strong instruction-following (IFEval 88.7) means reliable tool calling via Qwen-Agent. Keeps sensitive documents in-house.
Customer Support & Tier-1 Triage
Deploy as a private text classification & response-draft engine. Strong MMLU-Redux (93.1), reasoning (AIME25 70.3), and alignment (Arena-Hard v2 79.2) make it effective for intent detection, ticket summarization, and escalation decisions. No vendor lock-in; moderation and filtering happen in your environment.
Code Review & Engineering Workflow Automation
Leverage its coding ability (LiveCodeBench 51.8, MultiPL-E 87.9) to flag patterns, auto-document, or generate unit tests in CI/CD pipelines. Non-thinking mode (no <think> blocks) means fast inference for high-volume ops; outputs are deterministic and auditable.
Custom AI
As a base for custom AI
Strong foundation for bespoke internal tools: instruction-tuning on proprietary domain data (finance, legal, ops), embedding into custom UIs, or fine-tuning for specialized classification. Its broad capability across reasoning, multilanguage (MMLU-ProX 79.4), and tool use makes it adaptable to custom workflows without replacing the base model.
In the operating system
Where it fits
Sits at the inference core of an AI operating system: agent reasoning layer (tool calling, RAG decision-making), workflow automation backbone (document processing, triage), and knowledge layer (long-context retrieval over internal docs). Can power multiple departmental agents from a single shared private deployment.
Data control & security
Self-hosting eliminates data transmission to third-party inference APIs. Requests, outputs, and embeddings remain in your network and compute infrastructure. No model telemetry or usage tracking by default. You control access, logging, and retention policies. Note: self-hosting does not inherit security properties from the quantization or weights themselves; hardening is your responsibility (network segmentation, auth, audit logging).
Hardware footprint
Estimate (4-bit AWQ quantization): ~200–250GB VRAM total across 8× H100/A100 GPUs (tensor-parallel + expert-parallel). Single GPU inference not practical. Disk: ~116GB model download + overhead. Swap space recommended (model card specifies 16GB). Activation memory varies by batch size and context length.
Integration
Deploy behind an OpenAI-compatible REST API (vLLM native or via sglang ≥0.4.6.post1). Integrate via standard /v1/chat/completions endpoint into existing ops stacks (Zapier, n8n, custom apps). Use Qwen-Agent library for tool calling and MCP integration. Requires transformers ≥4.51.0 for qwen3_moe arch support. Context window up to 262K; truncate inbound prompts as needed.
When it's not the right fit
- —Single-GPU or edge-device deployment—this is a 235B model; no mobile/laptop inference.
- —Sub-millisecond latency required—LLM inference inherent; MoE routing adds overhead despite expert parallelism.
- —Thinking/reasoning transparency needed—non-thinking mode outputs only; no intermediate <think> blocks for explainability.
- —Real-time streaming to end-users at very high concurrency—vLLM handles ~512 seqs, but large-scale user-facing APIs may need additional load balancing.
Alternatives to consider
Llama-3.1-405B
Larger dense model, broader ecosystem, but no MoE savings; higher VRAM for same compute budget. Better if you need maximum density over efficiency.
DeepSeek-V3
Comparable MoE architecture with 671B total / 37B active; slightly lower reasoning benchmarks but proven in production. Check licensing (was previously restrictive).
Mixtral-8x22B-Instruct
Smaller MoE (176B total, 12.3B active), easier hardware footprint, but lower reasoning/math performance. Good if hardware is constrained.
FAQ
Can we run this fully private, no cloud at all?
Yes, if you own or lease 8 GPUs on-premises or in a private data center. vLLM runs in your environment; no calls to HF, Qwen, or any vendor occur after model download. Inference and outputs stay internal.
Is Apache-2.0 safe for commercial use in a private system?
Apache-2.0 permits commercial use, distribution, and modification, provided you retain the license notice. No warranty; you are liable. Review internally with legal if using modified versions or bundling derivatives.
How do we handle the 262K context in practice?
Use vLLM's --max-model-len flag. Model card defaults to 32K in examples to reduce memory. Longer contexts require proportionally more VRAM. Batch retrieval or hierarchical summarization for documents >32K tokens.
What's the difference between the base and this AWQ quantized version?
AWQ (4-bit) quantization reduces memory ~4× vs. fp16 with minimal accuracy loss, enabling 8-GPU multi-card inference. Base model requires more VRAM. Trade-off: decode speed vs. precision; Qwen reports this version is tuned to minimize degradation.
Ready to build a private AI operating system?
LLM.co helps mid-market teams self-host foundation models like Qwen3-235B for custom AI applications and operational automation—keeping your data and inference fully in-house. Let's architect your stack.