Open-Weight LLM · Private & Custom AI
Ornith-1.0-9B-MTP-GGUF
9B inference engine optimized for self-hosted private deployment with speculative decoding—run fast, quantized, on your own hardware.
Ornith-1.0-9B MTP is a Qwen3.5-9B hybrid fine-tune (by deepreinforce-ai) with a baked-in multi-token prediction head, distributed as GGUF quantizations for llama.cpp. A company runs it privately to reduce inference latency (1.38–1.73× speedup depending on precision) while keeping all text data in-house; the MTP head drafts multiple tokens in parallel, verified losslessly, so output quality is preserved.
Model facts
Private deployment
Run Ornith-1.0-9B-MTP-GGUF in your own environment
Deploy on any NVIDIA GPU (Ampere: use Q4_K_M 5.8 GB; Blackwell: NVFP4 6.6 GB for peak throughput at 306 tok/s). Load via llama.cpp or compatible runtimes (LM Studio, recent Ollama ≥0.31) with `--spec-type draft-mtp`. Data never leaves your network—architecture is purely on-premise, no external APIs. Requires llama.cpp ≥b9616 (Qwen3.5 arch + MTP support, shipped spring 2026). Low-VRAM variants (IQ2_M 3.9 GB, IQ3_M 4.7 GB) run on smaller GPUs or CPU with GPU acceleration.
Operational AI use cases
Customer Support Automation
Run a private, fine-tunable support chatbot on Ornith. MTP's speed (145+ tok/s on Q4_K_M) enables sub-second response drafting for ticket triage, FAQ routing, and sentiment analysis. All customer text stays in your systems—GDPR/HIPAA compliant by architecture. Customize the model weights for domain-specific tickets without model checkouts.
Document Processing & Internal Knowledge Extraction
Index and extract insights from internal docs, contracts, or operational logs using Ornith as the backbone. The 8192-token context window (with flash-attn) processes multi-page documents in one pass. Self-hosted means sensitive payroll, legal, or R&D docs never touch a third-party API. Attach MTP head via `--model-draft` to the base GGUF if you need to swap quantizations without re-training.
Workflow Agent & Task Automation
Build an autonomous agent (e.g., for finance reconciliation, supply chain alerts, or HR onboarding) that reasons over private data without external LLM calls. MTP's latency advantage (1.7× over non-speculative baseline) keeps agent loops tight. Deploy on a single GPU or CPU; quantized runtimes fit in 5–7 GB VRAM, so operations teams can spin up a dedicated inference pod without GPU cluster spend.
Custom AI
As a base for custom AI
Ornith is a strong base for custom AI products. The MIT-licensed trunk + head mean you own the weights and can fine-tune, quantize, and redistribute your own variants without licensing friction. Because it's Qwen3.5-9B under the hood, it inherits broad instruction-following and reasoning. Use the standalone MTP head (`mtp-head/mtp-Ornith-1.0-9B-head-Q8_0.gguf`) with your own quantized Ornith GGUF to compose a custom inference pipeline. The model card documents the grafting recipe—you can rebuild from source.
In the operating system
Where it fits
Ornith serves as the **inference core** in a private AI operating system: feed it into agent/workflow orchestration layers (ReAct, tool-use) via llama.cpp REST endpoints; couple it with your own vector DB (Pinecone, Weaviate) for retrieval-augmented generation; wire it into internal knowledge graphs for domain grounding. The MTP head is a performance multiplier, not a functional layer—it accelerates whatever reasoning or generation your ops workflows demand without changing outputs.
Data control & security
Self-hosted deployment means all inference data—customer text, internal docs, decision logs—stays within your network boundary. No data is sent to HuggingFace, OpenAI, or Anthropic. Compliance is an architectural property, not a model property: GDPR, HIPAA, SOC2 data residency is enforced by your infrastructure (VPC, air-gapped deployments) and your operational controls. The model itself is open-weight and auditable. Encryption, access logs, and audit trails are your responsibility; we recommend treating the Ornith inference pod like any other sensitive data system.
Hardware footprint
**Estimate**, assumes GPU VRAM for inference at batch size 1, context 8192, flash-attention on:** - Q4_K_M (Ampere recommended): ~6.8 GB VRAM - Q5_K_M (balanced): ~7.5 GB VRAM - Q6_K (near-lossless): ~8.2 GB VRAM - Q8_0 (reference quality): ~11 GB VRAM - NVFP4 (Blackwell): ~7.2 GB VRAM - IQ2_M (low-VRAM): ~5.5 GB VRAM (CPU+GPU mixed) CPU fallback: ~40–50 GB system RAM (slow, not recommended for ops use). Multi-GPU: split layers with `--n-gpu-layers` up to architecture depth.
Integration
Wire via llama.cpp's `/v1/chat/completions` (OpenAI-compatible API) or `/api/generate` REST endpoints. Docker/K8s: serve with `llama-server` in a sidecar container; scale horizontally with a load balancer and local VRAM quotas. CLI: single-line startup (`llama-server -hf protoLabsAI/Ornith-1.0-9B-MTP-GGUF:Q4_K_M --spec-type draft-mtp`). For workflows: call via Python SDK (llama-cpp-python), Node.js (node-llama-cpp), or any HTTP client. Existing ops tools (Zapier, Make, custom APIs) integrate via REST with no vendor lock-in. Bundle with LoRA adapters or prompt templates for domain tuning.
When it's not the right fit
- —**Retrieval or closed-domain QA under heavy volume**: MTP optimizes decode speed, not latency-sensitive classification. For real-time SLA'd support (sub-200ms response), consider distilled 3B models or a cached embeddings-first approach.
- —**Reasoning over massive structured data (SQL, analytics)**: Ornith is a text model. Use it for narrative summaries of query results, not as a SQL engine. Pair with deterministic tooling (query builders, formatters) for financial/ops data.
- —**Multi-modal workloads (images, audio, video)**: Ornith is text-only. If you need image-based docs or video transcripts, add a separate OCR or speech-to-text layer, then feed Ornith the text.
- —**Extreme privacy (e.g., military, classified)**: Self-hosting helps, but Ornith's weights derive from Qwen3.5 and other public data. If you need full IP/training provenance guarantees, audit the base model's training corpus or use a bespoke model trained on your data alone.
Alternatives to consider
Llama 3.2 9B (Meta, Hugging Face)
Similar size, broader instruction tuning, no MTP overhead. Slower decode (no speculative decoding built in), but stable and widely supported. Good if you don't need MTP's 1.7× speedup.
Mistral 7B Instruct (Mistral AI)
Smaller footprint (~4.2 GB Q4_K_M), faster base inference, commercial-friendly Apache 2.0 license. Trade: less powerful reasoning; choose if latency is critical and you can accept weaker outputs.
Phi-3.5 7B (Microsoft)
Specialized for instruction-following in a compact form; MIT license. Faster than Ornith-9B, but narrower domain coverage. Pick if your ops tasks are well-scoped (FAQ, simple routing) and you want minimal VRAM.
Related open models
FAQ
Can I run Ornith on my laptop or a single small GPU?
Yes. The IQ2_M quantization (3.9 GB) runs on a laptop GPU (RTX 4050, etc.) with 5 GB VRAM, though at slow speed (~20–30 tok/s). For better speed (100+ tok/s), a mid-range GPU (RTX 4060 Ti, A100) with 8–16 GB VRAM is practical. Q4_K_M (5.8 GB) is the sweet spot for Ampere GPUs.
Does Ornith stay under MIT license if I fine-tune or quantize it?
Yes. MIT is permissive—you can fine-tune, quantize, and redistribute derivative models under MIT without attribution (though we encourage it). Check the base `deepreinforce-ai/Ornith-1.0-9B` and MTP head repos for any additional terms; both are MIT.
What if I need the MTP speedup but want to swap the base model?
Use the standalone MTP head (`mtp-head/mtp-Ornith-1.0-9B-head-Q8_0.gguf`) with `--model-draft`. Pair it to any Ornith-9B GGUF base (different quantization, fine-tuned variant, etc.) via `llama-server --model-draft mtp-head/...`. Head is KL-distilled to Ornith's hidden states, so acceptance may degrade slightly on very different bases.
Is this model GDPR/HIPAA compliant?
No model is inherently compliant. Self-hosting Ornith *enables* compliance by keeping data in-house, but you must implement encryption, audit logging, access controls, and data retention policies in your infrastructure. Ornith is a tool; compliance is your responsibility.
Build a Private AI Operating System
Ornith is production-ready for self-hosted ops workflows. LLM.co helps you quantize, fine-tune, and integrate it into your ops stack—keeping data in your environment, no vendor lock-in. Start with a pilot: support automation, document processing, or autonomous agents. Let's talk.