Open-Weight LLM · Private & Custom AI
DeepSeek-V3.2-AWQ
685B parameter reasoning model optimized for private deployment via quantization (AWQ 4-bit), enabling companies to run sophisticated agentic AI and complex problem-solving workflows entirely within their own infrastructure.
DeepSeek-V3.2-AWQ is a 4-bit quantized version of DeepSeek's flagship 685B reasoning model, designed for on-premise execution via vLLM. It supports long-context reasoning, tool-use, and agentic workflows while keeping model weights and inference data under company control. For ops teams, this unlocks private automation of knowledge work, document analysis, and complex task orchestration without API dependency or data egress.
Model facts
Private deployment
Run DeepSeek-V3.2-AWQ in your own environment
Self-hosted via vLLM on NVIDIA H100/H800 clusters (Hopper architecture confirmed; Ada/Ampere untested). Requires CUDA 12.8, custom DeepGEMM kernels, and ~338 GiB disk storage. Setup demands tensor parallelism (8+ GPUs estimated for reasonable throughput) and careful environment configuration. Benefit: zero data leaves the company network; inference latency and costs become predictable capex, not recurring API spend.
Operational AI use cases
Internal Documentation & Knowledge Base Automation
Use the 32k context window to ingest entire policy documents, SOPs, or FAQs. Deploy as a private Q&A agent for HR, compliance, or finance teams. Long-context reasoning handles multi-document synthesis without external APIs; data stays internal.
Customer Support Triage & Escalation Logic
Reason over ticket histories and company-specific workflows to auto-classify support requests and recommend next steps. Tool-calling enables integration with internal ticketing systems. On-premise deployment ensures sensitive customer data never leaves your environment.
Contract & Invoice Analysis with Structured Output
Leverage reasoning capabilities to extract obligations, dates, and anomalies from procurement documents. Reasoning mode (disabled by default; enable via chat_template_kwargs) surfaces internal logic for audit trails. Chain-of-thought enables compliance-grade reasoning without vendor lock-in.
Custom AI
As a base for custom AI
Strong foundation for building proprietary AI products. 685B parameter scale and MoE sparse architecture support fine-tuning for domain-specific tasks (legal tech, biotech, supply-chain optimization). Tool-use framework and structured reasoning allow wrapping custom business logic into the model's decision flow. Private deployment means competitive differentiation stays in your codebase, not a third-party API.
In the operating system
Where it fits
Agent orchestration & reasoning layer. Sits between your workflow engine (task dispatch) and tool/integration layer (APIs, databases, internal services). Handles multi-step reasoning and tool selection; vLLM manages concurrency and memory. Pairs with an ops platform to manage prompts, logs, fine-tuning datasets, and model versioning.
Data control & security
Self-hosting eliminates data residency risk—prompts, contexts, and outputs never transit a vendor's servers. This is an **architecture choice**, not a model security feature. For compliance-sensitive work (healthcare, finance, PII-heavy ops), private deployment removes a class of data-leakage vectors. However: responsibility for securing the cluster, access control, and audit logging transfers to your team. Model itself has no built-in encryption or formal security certification.
Hardware footprint
**Estimate (unconfirmed for all setups):** AWQ 4-bit quantization reduces full-precision 685B to ~170–180 GiB active VRAM. Tested on H100 (Hopper); unclear on older architectures. With swap-space 16GB and 8× tensor parallelism, expect ~22–25 GiB per GPU. Single-GPU inference not viable; minimum cluster size 4 GPUs for latency-acceptable workloads.
Integration
Exposes OpenAI-compatible API via vLLM (port 8000 by default). Integrate via standard LLM SDKs or REST calls. Requires custom chat_template parsing (Python scripts provided in repo; not production-ready Jinja out-of-box). Tool calling uses deepseek_v31 parser; reasoning uses deepseek_v3 parser. Thinking mode off by default—enable per-request. Speculative decoding option (included in startup command) boosts throughput ~50% on compatible hardware.
When it's not the right fit
- —Real-time inference at scale without significant infrastructure investment: 8+ GPU cluster required; not a laptop/small-server model.
- —Chat template requires custom parsing; no off-the-shelf Jinja available yet. Production integration demands careful error handling and fallback logic.
- —Latency-critical <500ms responses: reasoning and MoE dispatch add overhead; speculative decoding helps but doesn't eliminate cold-start latency.
- —Frequent model updates expected: early release (Dec 2025); chat_template borrowed from v3.1 with thinking disabled by default; breaking changes possible.
Alternatives to consider
Llama 3.1 405B
Permissive license, broadly tested, smaller (~405B vs 685B). Better immediate vLLM support. Trade-off: less reasoning depth, no native tool-use framework, no sparse MoE.
Mixtral 8x22B
Proven MoE sparse architecture, smaller memory footprint, stable quantization support. Good for cost-conscious private deployment. Trade-off: lower reasoning performance, shorter context (32k vs unlimited claims).
Qwen2.5 72B
Lightweight, excellent tool-use, MIT-licensed, faster to self-host on modest clusters. Best fit for companies with <8 GPU budgets. Trade-off: significantly less reasoning capacity, no specialized agentic training.
FAQ
Can we fine-tune this model privately on our internal data?
Yes. MIT license permits modification. However: fine-tuning 685B requires significant compute (setup, custom DeepGEMM kernels, distributed training framework). Start with LoRA or prompt engineering. For full fine-tuning, factor in multi-week iteration and cluster provisioning.
Is this model safe for handling sensitive business data (PII, proprietary)?
Self-hosted deployment eliminates external data transfer—architecturally safer than API-based models. However: you own the burden of cluster security (access control, encryption-at-rest, audit logging, secure deletion). Model itself has no formal security audit. Data residency is your responsibility.
Can we use this commercially in a product we sell?
Yes. MIT license explicitly permits commercial use. You can sell products, services, or outputs built on DeepSeek-V3.2-AWQ. Requirement: include/preserve MIT license attribution in code/documentation. No royalty or approval needed.
What's the difference between the thinking_mode (on/off) and when would we use it?
Thinking (reasoning) mode is **off by default** in this quantized release. Enable per-request via chat_template_kwargs. Thinking exposes the model's reasoning chain (helps debug, improves compliance). Drawback: ~2–3× slower, longer token output. Use for high-stakes decisions (contract analysis, complex workflows); disable for speed-sensitive tasks (triage, rapid Q&A).
Build Private, Reasoning-Grade AI Into Your Operations
DeepSeek-V3.2-AWQ gives you a 685B reasoning engine that runs entirely in your environment. Partner with LLM.co to architect the infrastructure, fine-tune for your workflows, and operationalize agentic automation—without external API dependency or data leakage. Let's design your AI operating system.