Open-Weight LLM · Private & Custom AI
sarvam-30b-FP8-dynamic
FP8-quantized 30B multilingual model optimized for private inference—purpose-built for ops teams running proprietary workloads on controlled hardware.
Sarvam-30B-FP8-Dynamic is a quantized derivative of sarvamai/sarvam-30b, supporting 22 Indian languages plus English via dynamic FP8 quantization. It retains 99%+ accuracy of the unquantized base while cutting memory footprint by ~60%, making it viable for on-premise ops automation and custom AI at scale without cloud-dependency.
Model facts
Private deployment
Run sarvam-30b-FP8-dynamic in your own environment
Deploy via vLLM (validated on v0.18.0) on a single A100/H100 or multi-GPU via tensor parallelism. Data never leaves your environment—prompts, outputs, and logs stay internal. Requires vLLM + PyTorch + sufficient VRAM (estimate: ~16–20GB for FP8 @ full batch); Red Hat RHOAI/RHAIIS provide managed containerization if running on OpenStack/K8s. Private hosting eliminates inference telemetry and API call logging to third parties.
Operational AI use cases
Multilingual Customer Support Triage
Route and summarize support tickets in Hindi, Tamil, Bengali, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu, and English—keeping conversation history and customer PII on your servers. Use as backbone for in-house chatbot or ticket classification agent; no data leaves the data center.
Internal Knowledge Base & Process Documentation
Generate runbooks, FAQs, and process documentation from raw operational notes. Embed sarvam-30b-FP8 in a document-indexing pipeline (RAG) to surface internal procedures, troubleshooting steps, and compliance references without exposing proprietary workflows to external APIs.
Finance & Billing Automation
Extract entities (invoice dates, amounts, vendor names) from receipts and billing documents; flag anomalies or categorize expenses. Run entirely on-premise to maintain audit trails and regulatory compliance; integrate with ERP via REST / scheduled inference jobs.
Custom AI
As a base for custom AI
Lightweight base for fine-tuning on domain corpora (legal, medical, financial) across Indian languages. Apache 2.0 license permits commercial derivative models. Use LLM Compressor (open-source, included in creation recipe) to further customize quantization strategy or layer-specific pruning. Strong fit for building a proprietary copilot or vertical AI product with controlled cost-per-inference.
In the operating system
Where it fits
Sits at the **knowledge & reasoning layer** of an AI OS: feed it structured or semi-structured ops data (tickets, logs, documents) via a retrieval/RAG system, then route outputs to workflow automation (ticketing, CRM, ERP updates) or agent decision logic. Tensor parallelism scales it across multiple GPUs; vLLM's OpenAI-compatible API bridges it to standard orchestration layers.
Data control & security
Self-hosting means inference happens entirely within your network—no external API calls, no model outputs sent to vendors, no training data exfiltration risk. Compliance teams retain audit logs of all completions. Architecture choice isolates sensitive data (customer interactions, internal processes, financial records) from SaaS infrastructure. Note: This is a deployment property, not an inherent model guarantee; your infrastructure security depends on your own network, RBAC, and access controls.
Hardware footprint
**Estimate** (unverified): FP8 dynamic quantization ~16–18 GB VRAM on single A100 @ batch size 1–4; ~22–26 GB with tensor_parallel_size=2 on dual H100s. Full FP16/BF16 base (sarvamai/sarvam-30b) would require ~60 GB. Actual usage depends on context length (currently unknown), batch settings, and activation quantization overhead.
Integration
vLLM exposes an OpenAI-compatible `/v1/chat/completions` endpoint; integrate via standard HTTP clients or LangChain. Requires trust_remote_code=True (custom Sarvam tokenizer). Tensor parallelism flag (tensor_parallel_size) distributes across GPUs. Batch inference via `llm.generate()` for high-throughput ops (log analysis, bulk document processing). Validated on RHOAI 3.4, so containerize via Red Hat Ansible or Helm for K8s deployments.
When it's not the right fit
- —Context length is unknown—if your ops workload requires long document context (>8K tokens), test first or check base model specs.
- —You need real-time, sub-100ms latency: FP8 inference via vLLM is optimized but adds quantization overhead vs. GPU-native FP16; batch processing more efficient than single-token streaming.
- —Extensive code generation required: Sarvam is tuned for natural language and instruction-following; it is not a specialized code LLM (Codestral, DeepSeek-Coder may be better fits).
- —Your org exclusively uses non-Indian languages: while it supports English well, the architectural focus (22 Indian languages) means training data and optimizations may not suit specialized Western vertical domains equally.
Alternatives to consider
Llama 2 (70B, meta-llama/Llama-2-70b)
Larger, stronger English reasoning; less multilingual support. Requires ~140 GB VRAM unquantized. Good if you're English-only and can afford the inference cost.
Mistral-7B (mistralai/Mistral-7B-v0.1)
Smaller, faster, English-focused. Fits on smaller hardware (5–8 GB VRAM). Trade: weaker multilingual support and smaller context than sarvam-30b.
Aya-23-35B (CohereForAI/aya-23-35B)
Explicitly multilingual (101 languages), strong instruction-tuning. Similar memory footprint to sarvam-30b but different architectural choices; Apache 2.0 licensed.
FAQ
Can we run this on-premise without any cloud infra?
Yes—if you have a server with NVIDIA GPU (A100, H100, or RTX 6000+), vLLM runs standalone. No cloud required. You control all data ingress/egress and audit logs.
Is this model safe for commercial products?
Apache 2.0 permits commercial use, including derivative works. You may build a paid product on top of sarvam-30b-FP8-dynamic. Verify any specific trade compliance rules apply to your jurisdiction and end use.
How does FP8 quantization affect accuracy?
Benchmark data shows 92–101% recovery (e.g., GSM8K: 100.1%, IFEval: 92.6%). Slight drops on instruction-following tasks, negligible on reasoning/QA. Test on your ops data before production.
What if we need multi-language fine-tuning on proprietary domain data?
Apache 2.0 allows it. Use the unquantized base (sarvamai/sarvam-30b) for fine-tuning, then apply LLM Compressor to re-quantize. Custom derivatives remain proprietary; no obligation to publish.
Build Your Private AI Operating System Today
Sarvam-30B-FP8 is production-ready for on-premise ops automation. LLM.co helps you integrate it into a complete AI OS—knowledge retrieval, custom agents, workflow automation—keeping all data and IP under your control. Start building.