Open-Weight LLM · Private & Custom AI
Qwen3.6-35B-A3B-PRISM-NVFP4
Quantized MoE model for private, cost-efficient operational AI on NVIDIA Blackwell hardware, with softened refusal and bias mitigation built in.
Qwen3.6-35B-A3B-PRISM-NVFP4 is a W4A4 (FP4) quantized version of Qwen's 35B mixture-of-experts model, tuned to reduce over-refusal and remove propaganda patterns while preserving task performance. At ~24 GB on disk and designed for Blackwell tensor cores, it enables companies to run a capable, vision-aware LLM in a private environment without cloud dependencies. The MoE architecture (256 experts, ~3B active per token) trades off inference cost for accuracy, making it viable for high-volume operational workflows.
Model facts
Private deployment
Run Qwen3.6-35B-A3B-PRISM-NVFP4 in your own environment
Deploy via vLLM on Blackwell GPUs (H100/H200/H800 or DGX Spark) with the compressed-tensors quantization backend. Requires a patched vLLM build to support the SM120/SM121 NVFP4 kernels—community Docker images (e.g., ghcr.io/aeon-7/vllm-spark-omni-q36) eliminate manual PTX compilation. Data and model weights stay in your environment; no API calls. Estimated single-GPU footprint ~24 GB (quantized weights) plus activation memory (~20–30 GB during inference on moderate batch sizes). Multi-GPU serving with vLLM's pipeline parallelism is supported. This architecture eliminates vendor lock-in and ensures compliance-sensitive workloads (finance, healthcare, legal) never leave the corporate network.
Operational AI use cases
Conversational Support Automation & Knowledge Retrieval
Deploy as a RAG endpoint for internal support teams. Ingest proprietary docs (SOPs, policy manuals, customer knowledge bases) via vector DB + vLLM. The softened refusal behavior and multimodal capability allow it to handle image-based troubleshooting (e.g., equipment manuals, screenshots). Runs entirely on-prem; avoids exposure of internal documentation to external APIs. Spec decoding (with MTP draft head) reduces latency for synchronous chat interactions.
Operational Workflow & Process Automation
Use as a backbone for agentic workflows: contract analysis, invoice categorization, meeting summarization, incident triage. The 35B scale and MoE routing handle domain-specific language (legal terms, technical jargon) with high coherence. Self-hosted execution means your automation logic, extracted data, and decision trees remain proprietary. Integration with workflow engines (Zapier, Make, custom Python) via local API endpoint; no per-token SaaS cost at scale.
Internal Knowledge Base & Documentation Generation
Fine-tune or prompt-engineer for automated doc generation: runbooks, incident post-mortems, compliance reports, performance summaries. The bias/propaganda removal (PRISM tuning) reduces spurious outputs in sensitive contexts. Model runs in your DC; generated content can be staged, audited, and versioned without external cloud logs. Vision encoder preserved for table/chart extraction from scanned or digital documents.
Custom AI
As a base for custom AI
This is a strong foundation for custom AI applications because it's quantized, open-weight, and legally permissive (Apache 2.0). You can fine-tune on proprietary data (e.g., domain-specific chat, specialized code generation, industry jargon) without licensing friction. The MoE structure and preserved multimodal capability suit vertical applications (e.g., financial analysis with document ingestion, healthcare admin with image processing). Because it runs privately, you retain all training data and model derivatives. Competitors' proprietary APIs cannot match the cost-per-inference or data sovereignty at this scale.
In the operating system
Where it fits
In an LLM.co-style ops AI stack: **Knowledge Layer** (retrieval, doc understanding via vision encoder); **Workflow/Agent Layer** (reasoning, task decomposition, tool orchestration via conversational interface). It is not ideal as a cheap embedding model or specialized classifier—use smaller models for those. It excels as the central reasoning engine for multi-step operational tasks, combining language understanding with optional visual context. Pair it with a vector DB (for RAG), a workflow orchestrator (for agents), and a local model API (vLLM) to route requests without cloud egress.
Data control & security
Self-hosting this model ensures data residency: queries, retrieved documents, and inference outputs remain in your infrastructure and never transit to external vendors. This is an architectural benefit—the model itself is not inherently 'secure' or 'compliant,' but deployment topology enables you to build secure systems (VPC isolation, audit logging, encryption at rest/transit) that meet regulatory requirements (SOC 2, HIPAA, GDPR). No external telemetry or model callbacks; you control access logs and inference history. For compliance-sensitive use cases, conduct your own security review and penetration testing; Qwen/Ex0bit provide no formal security audit or SLA.
Hardware footprint
**Estimate (W4A4 quantization):** ~24 GB model weights (disk/main memory). **Inference peak:** ~30–50 GB VRAM (including activations + KV cache, batch size 1–16, context ~8k tokens). **VRAM by precision (reference, not this quantization):** full BF16 ≈ 140–160 GB; bfloat16 + NF4 quantization ≈ 40 GB; W4A4 ≈ 24–30 GB. Minimum viable GPU: NVIDIA H100 (80 GB HBM3, max batch ≈16–32). Dual H100 or DGX Spark (8× GB10) for production workloads with overlapped inference + batching. Power envelope ~400–500W (H100); cooling and infra cost should be factored into TCO vs. cloud APIs.
Integration
vLLM exposes OpenAI-compatible REST API (start with `vllm serve`); integrate via standard libraries (e.g., `openai` Python client, `curl`) or LangChain/LlamaIndex. Chat template (`chat_template.jinja`) is included; no template engineering needed. MTP draft head (speculative decoding) is optional—use `model_mtp.safetensors` for lower latency if supported by your vLLM build. Tokenizer files are pre-packaged. For agentic workflows: wrap the API endpoint with a tool-calling orchestrator (e.g., LangChain agents, custom Python loops) and define tool schemas in your prompt. Vision encoder is enabled by default; pass images in multimodal format (if your integration layer supports it). Batch inference via vLLM's batching; no special config needed.
When it's not the right fit
- —You need sub-100ms latency at scale without speculative decoding and custom kernel tuning. MoE routing overhead + FP4 conversion adds ~50–150ms per token on Blackwell vs. dense models.
- —You require strong performance on code generation or math. Qwen3.6 is general-purpose; specialized code/math models (DeepSeek-Coder, etc.) will outperform. PRISM tuning may further reduce edge-case code reasoning.
- —Your ops team lacks NVIDIA GPU infrastructure or DevOps expertise to maintain vLLM, Docker builds, and kernel patches. The Blackwell requirement and custom PTX instructions impose setup friction.
- —You need real-time <1-second response in a high-concurrency setting (>100 req/sec). MoE routing latency compounds; consider smaller dense models or quantized Llama variants for throughput-critical tasks.
Alternatives to consider
Meta Llama 3.1-405B (quantized, e.g., GGUF or GPTQ)
Larger, denser, stronger at reasoning and code. Runs on CPU (GGUF) or older GPUs (GPTQ). Less specialized for ops bias/refusal mitigation; requires more VRAM/inference cost per token.
DeepSeek-V3 (MoE, open-weight, Apache 2.0)
Also MoE, also quantizable, strong reasoning and Chinese NLP. Requires SGLang/vLLM support; ecosystem is less mature than Qwen. Similar VRAM footprint, different tuning philosophy (no PRISM bias removal).
Mistral Large (open-weight, ~47B, or NV-Mistral quantized)
Slightly smaller active compute, strong on instruction-following and ops tasks. Better latency/throughput than 35B MoE on single-GPU. Less multimodal; no PRISM tuning; different license (Mistral Research License, review for commercial use).
Related open models
FAQ
Can I run this on a single A100 or RTX 6000?
No. This model requires NVIDIA Blackwell (H100, H200, H800, or DGX Spark GB10) for the SM120/SM121 NVFP4 kernels. Older architectures (A100, RTX 6000) do not have the FP4 E2M1 hardware support; you would need a software fallback (slower) or a different quantization (e.g., GPTQ, NF4).
Is Apache 2.0 license truly commercial-use permissive?
Yes. Apache 2.0 permits commercial use, modification, and distribution with minimal restrictions (retain license notice, state changes). You can fine-tune, sell products using this model, and keep derivatives proprietary. No royalties or per-inference fees to Ex0bit or Qwen. Review the license text for edge cases (patent claims) if you have legal counsel.
What's the difference between PRISM tuning and the base Qwen3.6-35B-A3B?
PRISM softens over-refusal behavior and removes detected bias/propaganda patterns while maintaining task performance. In ops context: it reduces false 'I cannot help with that' responses on legitimate internal tasks (e.g., policy summaries, data analysis). It is not 'jailbreak tuning'—it targets reducing spurious safety interventions. You inherit the base Qwen safety alignment; PRISM refines it. Review outputs in your domain before production.
Can I use the MTP draft head for speculative decoding?
Yes, if your vLLM build supports speculative decoding (community images do). Load `model_mtp.safetensors` alongside the main model. This runs a smaller draft model in parallel to predict the next token(s), reducing latency by 20–40% on long-output tasks. Standard vLLM serving (without spec decode) also works; draft head is optional.
Build Custom Ops AI Without Vendor Lock-In
This model is ready for self-hosted deployment, but integrating it into your existing systems, fine-tuning it on proprietary data, and scaling it across your ops workflows requires orchestration. LLM.co helps you build the full stack: private LLM serving, RAG integration, workflow automation, and custom AI applications—all in your environment. Let's explore how Qwen3.6-35B can power your internal knowledge and agent layers.