Open-Weight LLM · Private & Custom AI
Qwen3.5-122B-A10B-heretic-MTP-NVFP4
Quantized 122B MoE model optimized for high-throughput private inference on enterprise hardware—speculative decoding and tensor parallelism built in for ops automation at scale.
Qwen3.5-122B-A10B-heretic-MTP-NVFP4 is a W4A4 (FP4) quantized Mixture-of-Experts model with ~10B active parameters, multimodal (vision + text) capability, and working speculative decoding via MTP. For ops teams, it trades a fraction of reasoning quality for dramatic cost and latency reduction—fit for high-volume internal automation (support triage, document processing, knowledge extraction) that runs entirely within your firewall.
Model facts
Private deployment
Run Qwen3.5-122B-A10B-heretic-MTP-NVFP4 in your own environment
Runs on 2x consumer-grade Blackwell or enterprise GPUs (RTX 6000 Pro tested; ~76GB disk, estimated ~32–48GB VRAM per GPU with TP=2). Deploy via vLLM 0.19+, which handles tensor parallelism, prefix caching, and MTP speculative decoding out of the box. No external inference service required; all data stays in your data center. Calibration is CPU-intensive (~400GB swap, 3 days) but one-time; inference is production-ready post-quantization.
Operational AI use cases
Support Ticket Auto-Triage & Routing
Classify and extract intent from support tickets in real time, suggest routing to specialists, and draft initial responses—all within your own environment. Multimodal capability handles screenshots and PDFs. 190 tok/s throughput (with MTP enabled) processes hundreds of tickets per hour per GPU.
Internal Knowledge & Policy Q&A
Build a RAG layer atop this model to answer employee questions about HR policy, IT procedures, finance rules, or product docs without leaving your infrastructure. Vision support means indexing internal diagrams, org charts, and reference materials.
Agentic Document Automation
Extract structured data from contracts, invoices, POs, and reports; flag anomalies; and trigger downstream workflows (procurement, billing, compliance). Runs as a local agent—no cloud API keys, no data exfiltration, no per-token billing.
Custom AI
As a base for custom AI
Strong baseline for building proprietary ops AI applications. The abliterated heretic variant and quantization preserve reasoning while dropping cost; you can fine-tune on your own ops-specific data (tickets, docs, workflows), quantize further if needed, and ship as a private model. Multimodal + MoE + speculative decoding mean you have a sophisticated backbone without the overhead of a full 122B model—ideal for white-label or internal product builds.
In the operating system
Where it fits
Sits at the center of the **workflow execution layer** in an ops AI OS—the brains behind agents that automate support, knowledge, and document tasks. Feeds into agentic orchestration (tool calling, reasoning loops) and connects to your CRM, knowledge base, and automation platform via vLLM APIs. Knowledge layer provides context via RAG; workflow layer triggers actions based on model output.
Data control & security
Self-hosting eliminates data transit to third-party APIs—sensitive employee data, customer info, contracts, and policies remain in your VPC/data center. You control access logs, fine-tuning datasets, and model updates. Quantization reduces attack surface (smaller model = smaller memory footprint) but does *not* inherently encrypt or isolate data; network segmentation and encryption-in-transit remain your responsibility. No license-based privacy guarantees—this is an architecture choice, not a security property of the model itself.
Hardware footprint
**Estimate (W4A4 quantization):** ~32–40 GB VRAM per GPU (TP=2, batch size 1–4). Disk: 76 GB (quantized model + shards + vision encoder). Tested on 2x RTX 6000 Pro Blackwell; likely runs on single A100/H100 80GB with TP=1, though throughput drops. **Calibration overhead:** ~400 GB swap + 64 cores CPU for 3 days one-time.
Integration
vLLM REST API or OpenAI-compatible endpoints map directly to LangChain, LlamaIndex, or custom Python/Node agents. Tensor-parallel setup (TP=2 recommended) requires orchestration (Docker, K8s) to manage shards across GPUs. Tool-calling parser (`qwen3_coder`) integrates with function-calling workflows. Prefix caching reduces latency for repeated context (e.g., RAG chunking). Export via `endpoints_compatible` tag means it works with HF Inference API if you ever need serverless fallback.
When it's not the right fit
- —Multi-turn reasoning at frontier-model quality—abliterated heretic + quantization trade off some nuance vs. base Qwen3.5. If your use case demands GPT-4 reasoning, look elsewhere.
- —Extremely latency-sensitive serving (<50ms p99 for single requests)—quantization helps, but MoE routing adds overhead. Pure inference speed beats this on smaller, unquantized dense models.
- —Your ops team lacks GPU infrastructure or Kubernetes experience—deployment is not trivial; vLLM + tensor parallelism requires DevOps lift.
- —Highly regulated compliance scenarios (finance, healthcare) where model provenance, audit trails, and formal validation are required—open-weight models and quantization chains lack formal approval paths.
Alternatives to consider
Meta Llama-3.1-405B (quantized via GGML/gptq)
Larger, denser, less quantized—better quality reasoning but much higher VRAM (fits 70B or smaller variants on same hardware). Slower inference, no speculative decoding out of box.
Mistral Mixtral-8x22B (Q4 quantized)
Smaller MoE (8x22B vs 122B), lower memory, similar throughput. Lacks vision and speculative decoding; good for pure-text ops tasks if you don't need multimodal.
DeepSeek-V3 (if open-weight release happens)
Reported to be more efficient than Qwen3.5 at similar scale. Watch for release; likely to have better reasoning-per-FLOP. Current status: Unknown / proprietary.
Related open models
FAQ
Can we run this entirely on-premises with no external APIs?
Yes. Deploy via vLLM on your own GPU cluster (2x RTX 6000 Pro or equivalent). Data never leaves your VPC. You own the deployment, logs, and fine-tuning pipeline. No licensing restrictions on commercial or internal use (Apache 2.0).
Is this model commercially usable, or do we need Qwen's permission?
Apache 2.0 license permits commercial use outright—no permission needed. The heretic abliteration and quantization are also Apache 2.0. You can embed it in products, sell services powered by it, and fine-tune freely. No royalties, no usage caps.
How much faster is it with speculative decoding enabled?
With MTP enabled (6 speculative tokens), measured throughput jumps from ~105 tok/s (disabled) to ~190 tok/s—an **81% improvement** on 2x Blackwell. Real-world gain depends on your workload; document summarization benefits more than single-turn QA.
Can we fine-tune this model on our internal data?
Yes, but requires careful setup. Quantized models need quantization-aware fine-tuning (QAT) or unquantized base-model tuning + re-quantization. Toolkit: `llm-compressor` (as used here) or LoRA-based approaches. Start small; calibration is expensive.
Build Your Private Ops AI Stack
Ready to run this (or another open-weight model) in your own data center? LLM.co helps enterprises architect private LLM deployments, set up vLLM + orchestration, and build custom AI agents that stay within your firewall. Schedule a review of your ops bottlenecks—we'll size the right model and GPU setup for your workflows.