Open-Weight LLM · Private & Custom AI
Gemma-4-26B-A4B-it-NVFP4
Production-ready quantized MoE model for private, cost-efficient inference on modern hardware—deploy multi-billion-parameter reasoning in your own environment.
Gemma-4-26B-A4B-it-NVFP4 is a W4A4-quantized version of Google's Gemma 4 Mixture-of-Experts model, compressed 3x to 16.5 GB while retaining 97.6% benchmark quality and doubling throughput. For ops teams building private AI systems, it delivers the reasoning capability of a 25B-parameter model at the resource footprint and speed of a much smaller system—no API calls, no data leaving your infrastructure.
Model facts
Private deployment
Run Gemma-4-26B-A4B-it-NVFP4 in your own environment
Run it fully self-hosted via vLLM on Nvidia GPUs (tested on DGX Spark/Blackwell; works on A100/H100 clusters with appropriate quantization kernels). Requires vLLM ≥5.4 with transformers support, the included gemma4_patched.py for MoE expert routing, and marlin kernel backend. Data stays entirely in your environment; model loads at ~15.7 GB on DGX Spark, leaving room for batch inference and long contexts (256K). Operationally: one container, no external calls, full API compatibility (OpenAI-compatible chat endpoint).
Operational AI use cases
Internal Support Agent & Knowledge Automation
Deploy as a private chatbot for HR, IT, or legal document Q&A. Instruction-following (IFEval 98%+ retained) means it accurately answers policy questions, escalates edge cases, and reduces first-contact resolution time. Multi-modal (text + image) supports doc scanning and form parsing. No external API = compliance-ready for regulated data.
Financial & Operational Reporting Automation
Feed internal reports, ledgers, and unstructured logs into the model for summarization, anomaly flagging, and narrative report generation. 256K context handles full quarter statements. ~4pp quality drop in math reasoning (GSM8K 87.79%→84.23%) is acceptable for summarization and pattern-finding; finance teams validate outputs anyway. 2x throughput vs. BF16 means daily batch jobs complete in half the time.
Workflow Automation & Agent Backbone
Use as the reasoning engine for multi-step ops workflows: ticket triage (classify → route → escalate), procurement workflows (request parsing → policy check → approval), or incident response (alert parsing → runbook recommendation → human handoff). MoE architecture activates only 3.8B/25.2B parameters per token—efficient for high-volume agentic calls. Low latency (TTFT 53ms) supports real-time decision loops.
Custom AI
As a base for custom AI
Strong base for building proprietary AI products atop open weight. W4A4 quantization is lossless enough for fine-tuning (math drops ~4pp; instruction-following near-lossless). If you're building a domain-specific support assistant, internal knowledge bot, or agentic workflow, you can fork this checkpoint, add LoRA adapters or continue-pretrain on private domain data, and ship a production model that never phones home. Apache 2.0 license permits commercial use and derivative products.
In the operating system
Where it fits
Knowledge/reasoning layer of a private AI operating system. Sits downstream of data ingestion (vector DBs for RAG context), upstream of workflow orchestration (tool-calling → execution). The MoE architecture makes it economical to run continuously (agentic loop friendly). Pair with retrieval, structured output tools, and state management for multi-turn agentic ops.
Data control & security
Self-hosting means all prompts, context, and model outputs remain in your data center—no logs on external servers, no third-party API providers. This is an architecture choice, not a claim about the model itself. Compliance teams appreciate the audit trail: inference happens on your infrastructure, under your access controls. Quantization does not reduce security; it only reduces VRAM footprint. Treat deployment security per your infrastructure standards (network isolation, RBAC, audit logging).
Hardware footprint
**Estimate (W4A4).** Model weights: 16.5 GB on disk. VRAM at inference (single sequence): ~18–20 GB (weights + KV cache + activations). Batched inference (4 concurrent requests, 256K max context): 22–26 GB with FP8 KV cache. Originally 49 GB BF16 → 3x compression. Tested on DGX Spark (128 GB unified); fits comfortably on dual H100 (80 GB each) or A100 (80 GB) with room for context.
Integration
Deploy via Docker + vLLM on Kubernetes or standalone GPU nodes. Exposes OpenAI-compatible /v1/chat/completions endpoint—plug into any LLM framework (LangChain, CrewAI, llama-index, custom Python). Batch inference via vLLM's async API for high-throughput ops tasks. Vision encoding stays in-model (no separate CLIP); image inputs work natively. Requires `--trust-remote-code` flag. Marlin kernel dependency means vendor lock-in to Nvidia hardware; port to AMD/Apple silicon Unknown. Post-quantization reproducibility documented via quantize_gemma4_moe.py script.
When it's not the right fit
- —Math-heavy applications where ~4% accuracy loss matters: GSM8K drops 87.79% → 84.23%. Fine for summarization; risky for financial calculations without post-validation.
- —Latency-critical single-token tasks (<10ms TTFT required): vLLM overhead + quantization overhead gives 53ms TTFT—good for batch and agentic loops, not for ultra-low-latency single-turn chat.
- —Non-Nvidia GPU deployments: Marlin kernel backend is Nvidia-only (A100, H100, Blackwell, etc.). ROCm/Metal/other accelerators would require re-quantization or kernel porting (Unknown effort).
- —You need guaranteed API compatibility with all LLM frameworks: W4A4 + MoE expert routing requires patches (included gemma4_patched.py); not all frameworks support it out-of-box.
Alternatives to consider
Llama 3.1 70B (BF16 or GGUF-quantized)
Larger, denser model (no MoE) with higher absolute quality. Not quantized at source, so you manage quantization yourself. Better for dense reasoning; heavier on VRAM (requires A100 or larger).
Mixtral 8x22B (MoE, BF16 or similar quantizations)
Comparable MoE architecture, 141B total / 39B active. More experts, higher quality, but larger footprint (~90 GB unquantized). If you have cluster VRAM to spare and want stronger performance.
Qwen2.5 32B (BF16 or quantized variants)
Denser, non-MoE alternative with strong instruction-following and multilingual support. Simpler deployment (no expert routing), but smaller window (128K vs. 256K). Better if you prefer predictability over MoE efficiency.
Related open models
FAQ
Can I run this on my own GPU clusters without touching Nvidia's APIs?
Yes. vLLM + this model + your Nvidia hardware (A100, H100, DGX, etc.) = fully private. Download the model from HuggingFace once, load in vLLM, expose an OpenAI-compatible endpoint on your internal network. Data never leaves your data center. Requires ~16–22 GB VRAM per inference instance.
What about commercial/production use? Can we build a product on this?
Apache 2.0 license explicitly permits commercial use, derivative works, and commercial products. You can fine-tune it, ship it in a product, monetize it. No royalties to Google or bg-digitalservices. Verify compliance with your legal team for any restrictions in your jurisdiction.
How much faster is this vs. the original BF16 Gemma 4?
~2.07x throughput on DGX Spark: 48.2 tok/sec (W4A4) vs. 23.3 tok/sec (BF16). TTFT cut from 97ms to 53ms. Trade-off: math reasoning loses ~4% (within acceptable range for summarization/classification; validate critical calculations separately).
What if I want to fine-tune this model on my proprietary data?
Technically feasible but requires care. W4A4 quantization is post-training; unfreezing and re-training on quantized weights can degrade results. Common approach: load the original BF16 Gemma 4, apply LoRA adapters to your data, then quantize the LoRA output. Or use the quantized model for inference only and train a separate BF16 head. Contact LLM.co for guidance on your use case.
Build Private AI Into Your Operations
Gemma-4-26B-A4B-it-NVFP4 is proof that you don't need closed APIs to power intelligent workflows. Whether you're automating support, building agentic ops systems, or deploying compliance-ready knowledge bots—own your model, own your data. Let LLM.co help you integrate quantized, open-weight LLMs into your ops stack. Talk to us about building your AI operating system.