Open-Weight LLM · Private & Custom AI
Moonlight-16B-A3B-Instruct
A 16B-parameter Mixture-of-Experts model that activates only 3B parameters per inference, designed for cost-efficient private deployment in ops automation and custom AI applications where inference speed and resource footprint matter.
Moonlight-16B-A3B-Instruct is an instruction-tuned MoE model trained with the Muon optimizer on 5.7T tokens, achieving strong benchmark performance (MMLU 70.0, HumanEval 48.1) while requiring ~2× fewer FLOPs than comparable dense models. For ops teams, this means deploying a capable reasoning engine on modest hardware while maintaining full data privacy in self-hosted environments.
Model facts
Private deployment
Run Moonlight-16B-A3B-Instruct in your own environment
Self-hosting Moonlight means running a 16B-parameter model that only activates ~3B per token, reducing memory and compute demands vs. dense alternatives. A company controls the full inference pipeline—no model telemetry, no API calls—making it suitable for sensitive workflows (finance, legal, internal KM). Estimate ~32GB VRAM (bfloat16) for a single GPU; can shard across multiple devices. Supports vLLM and SGLang for production serving. Trade-off: you own ops (monitoring, versioning, fine-tuning infrastructure).
Operational AI use cases
Internal Knowledge & Policy Q&A (Support Triage)
Deploy Moonlight to answer employee and customer questions against private docs (employee handbook, product guides, FAQ). Use RAG to ground responses in company-owned knowledge. MoE efficiency means per-query cost stays low even under high concurrency. Runs entirely in your VPC—no external API calls, no data leakage.
Contract & Document Review (Legal/Finance)
Build a document classifier and summarizer for inbound contracts, NDAs, and invoices. Moonlight's code and reasoning performance (HumanEval 48.1, MATH 45.3) supports extraction of payment terms, risk flags, and obligation mapping. Self-hosted means zero exposure of proprietary contract language to third parties.
Ticket Routing & Escalation (Ops Workflow)
Automatically categorize incoming support tickets, assign severity, suggest a first response draft, and flag for escalation. MoE activation reduces latency for high-volume intake. Integrate with Jira, Zendesk, or internal ticketing via REST—train on anonymized ticket history to refine routing over time.
Custom AI
As a base for custom AI
Moonlight is a strong base for building proprietary vertical AI products (industry-specific assistants, domain-tuned classifiers) because it's lightweight, open-weight, and fine-tunable. Its MoE architecture allows selective task-specific expert activation; teams can further specialize experts or add domain adapters. MIT license means commercial products built atop it are permissible—no royalty or attribution complexity.
In the operating system
Where it fits
In an AI Operating System, Moonlight anchors the **reasoning/agent layer** below orchestration. It's the 'brain' for multi-step workflows: it ingests structured + unstructured data (documents, logs, chat), reasons over it, and returns actions (classifications, summaries, recommendations). Layer it with a retrieval system (vector DB) for context, a workflow engine (state machine) to chain operations, and a compliance layer (audit, redaction) for regulated use.
Data control & security
Self-hosting Moonlight means your inference data (user queries, documents, context) never leaves your infrastructure—critical for regulated industries and proprietary knowledge. No model calls home; no third-party logging or behavioral tracking. That said, the model itself is not 'secure by design'—you must apply standard hardening (network isolation, access controls, input validation, output filtering). Security posture depends on your deployment architecture, not the model.
Hardware footprint
**Estimate (bfloat16):** ~32 GB VRAM for single-GPU inference; ~16 GB per GPU if sharded across 2 devices. Pretrained model uses safetensors format (~32GB disk). At inference, only ~3B parameters are active per token, so throughput scales well. For production: RTX 6000, A100 40GB, or H100 are suitable; multi-GPU setups recommended for >50 req/s.
Integration
Moonlight accepts standard transformers / text-generation-inference APIs (compatible with vLLM, SGLang, TGI). Expose it as a REST endpoint or gRPC service in your ops stack. Integrate via Langchain, LlamaIndex, or custom agents for retrieval + context-building. Supports batching for multi-query workloads. For real-time ops dashboards, ensure monitoring (latency, throughput, error rates); log inference metadata (not user data) for optimization.
When it's not the right fit
- —Your ops workflow requires <50ms latency end-to-end and you lack GPU infrastructure; even optimized, MoE routing adds overhead vs. dense models.
- —You need multilingual support beyond English/Chinese; Moonlight's training data and evals are EN/ZH focused; other languages require in-house adaptation.
- —You require a fully commercial, off-the-shelf managed service with SLAs and liability; self-hosting means you own availability and correctness.
- —Your use case demands extremely small context (Moonlight is 8K tokens); models like Phi-4 or Qwen-0.5B may be more efficient for ultra-constrained deployments.
Alternatives to consider
DeepSeek-V2-Lite (2.4B activated / 16B total)
Nearly identical architecture, also MoE. Moonlight outperforms on MMLU (70 vs 58.3) and code tasks; V2-Lite is slightly better on GSM8K. Choose V2-Lite if you need proven production track record; Moonlight if you prioritize latest research & FLOP efficiency.
Qwen2.5-3B (dense, 3B parameters)
No MoE overhead, smaller memory footprint (~6GB bfloat16). Better on GSM8K (79.1 vs 77.4). Faster inference, lower latency, but same total throughput on same GPU. Pick Qwen if latency is critical; Moonlight if you want faster tokens/sec or finer task specialization via MoE.
Llama-3.2-3B (dense, 3B parameters)
Proven ecosystem, excellent community support. Weaker on benchmarks (MMLU 54.75, HumanEval 28.0). Simpler to fine-tune. Use if you heavily rely on existing Llama tooling or need long-tail compatibility; Moonlight if raw capability and FLOPs matter more.
Related open models
FAQ
Can I fine-tune Moonlight privately on my company's data?
Yes. MIT license permits modification. You can run LoRA or full fine-tuning on your own infrastructure. Expect ~40–80GB VRAM for full training; LoRA is more practical on single GPUs. Store checkpoints locally; no external training APIs needed. You own all weights.
What are the commercial-use terms?
MIT license allows commercial products, redistribution, and modification with no attribution or royalty requirements. You can build a SaaS or closed-source product on Moonlight without legal friction. Review your legal team's interpretation if highly regulated (healthcare, finance); license is permissive but doesn't grant liability indemnity.
Is Moonlight better than just using ChatGPT via API?
Trade-offs: Moonlight is cheaper per inference, keeps data private, and runs fully under your control—critical for IP or regulated workloads. ChatGPT is more capable (reasoning, long context, multimodal), fully managed, and requires no ops. Use Moonlight for private/cost-sensitive; ChatGPT for best-in-class reasoning and minimal DevOps.
What's the typical latency for an inference request?
Unknown from model card. Depends on hardware (GPU tier), batch size, context length, and inference engine (vLLM vs. TGI). Single-token latency ~50–200ms on A100 is typical for MoE. Run benchmarks in your target environment; MoE routing adds ~5–10% overhead vs. dense models.
Build Private AI Into Your Ops Stack
Moonlight is primed for self-hosted deployment—but integrating it into live workflows requires orchestration, monitoring, and fine-tuning. LLM.co helps you architect a private AI OS: embed Moonlight into document review, ticket automation, or knowledge search. Let's scope your first ops-AI use case.