Open-Weight LLM · Private & Custom AI
Ornith-1.0-35B-NVFP4-MTP-GGUF
A 35B Mixture-of-Experts reasoning model optimized for private Blackwell GPU deployment, combining MXFP4/NVFP4 quantization with speculative decoding—built for companies running closed-loop agentic workflows without model exfiltration.
Ornith-1.0-35B is a Qwen3.5-MoE derivative post-trained via RL for agentic coding and reasoning, quantized in two variants (MXFP4 or NVFP4) for NVIDIA Blackwell GPUs with grafted multi-token prediction heads enabling ~1.5–2× decode speedup. For ops teams, it delivers reasoning-quality inference (reasoning_content block + draft verification) in a format that runs entirely on-premise in 19–20 GB VRAM, keeping IP and operational data locked in your environment.
Model facts
Private deployment
Run Ornith-1.0-35B-NVFP4-MTP-GGUF in your own environment
Runs via llama.cpp (GGUF format) on Blackwell hardware (RTX PRO 6000, DGX Spark). The GGUF quantization is pre-compiled; no further tuning needed. Companies deploy this on isolated GPU clusters, air-gapped networks, or private cloud infrastructure (AWS private instances, on-prem data centers). Measured on RTX PRO 6000: 240 tok/s baseline, 310–320 tok/s with MTP—acceptable for batch-mode reasoning work. On DGX Spark (3-slot parallelism): ~31 tok/s per slot. All model weights, KV cache, and intermediate activations stay within your hardware boundary; no telemetry, no third-party inference endpoints.
Operational AI use cases
Autonomous code review & bug triage
Route incoming GitHub issues, PRs, and test failures to Ornith running locally. The reasoning_content block surfaces decision logic; reasoning + agentic-coding training let it propose fixes, rank severity, or auto-escalate. Output stays inside your VCS and never touches external APIs.
Internal knowledge synthesis & search augmentation
Index company docs, runbooks, past tickets, and architecture diagrams into a RAG pipeline backed by Ornith. Ops/SRE queries ("why did we choose Postgres over MySQL?" or "how do we handle failover?") get grounded, reasoned responses without shipping your knowledge base to a third party.
Contract & compliance document classification
Run Ornith in batch mode on legal agreements, SOWs, and NDAs. The agentic-coding training + reasoning head let it extract obligations, flag red flags, and categorize contract type—all on-premise, satisfying audit and data residency constraints in Finance/Legal workflows.
Custom AI
As a base for custom AI
Strong candidate for a fine-tuning base. Ornith inherits Qwen3.5-MoE's 35B total / ~3B active parameter shape, making it RAM-efficient for custom instruction tuning on Blackwell. The MIT license permits retraining and redistribution. You can graft domain-specific reasoning heads (similar to how MTP was added) or LoRA-adapt the router for your vertical (e.g., medical coding, financial triage). The GGUF format is frozen; custom training requires converting back to HF format (reversible via llama.cpp tools), then re-quantizing.
In the operating system
Where it fits
Sits at the **reasoning + agent execution layer** of a private AI OS. Upstream: embedding / retrieval pipelines (for RAG context). Downstream: workflow automation (routing, approval, escalation logic) and compliance gates. The multi-token prediction head acts as an internal draft mechanism, so the model can serve both interactive (chat) and batch (async reasoning) patterns without external APIs.
Data control & security
Self-hosting Ornith means all prompts, reasoning traces, and outputs remain in your physical or logical network perimeter. No data is sent to HuggingFace, OpenAI, or any third-party service—your confidential code reviews, internal docs, and contracts never leave your infrastructure. This is an *architectural* choice, not a property of the model itself: the model is agnostic to data sensitivity. Compliance benefit: you control encryption-at-rest (VRAM, disk, network), audit logging, and access controls. Limitation: you own security of the Blackwell GPU cluster and llama.cpp runtime—patches, VRAM isolation, and key management are your responsibility.
Hardware footprint
**Estimate (not verified in your environment):** - **NVFP4 variant (19.85 GiB model weights):** ~24–26 GB total VRAM (weights + KV cache at 200k context on RTX PRO 6000). Single-slot deployment: RTX PRO 6000 (97 GB GDDR7) sufficient. - **MXFP4 variant (20.30 GiB model weights):** ~24–27 GB total VRAM; footprint essentially identical to NVFP4 at inference. - **Multi-slot (DGX Spark):** 3 × 121 GB unified LPDDR5X (total 363 GB). Practical limit: 3 parallel 200k-context slots (not tested at 600k combined). - Bandwidth-bound on both platforms; CPU offload (using system RAM) possible but not recommended (>10× slowdown observed).
Integration
Ornith runs via llama.cpp, which exposes HTTP/REST endpoints (llama-server) compatible with OpenAI-style chat APIs. Drop it into your ops platform (Zapier, internal Python automation, or custom agent framework) as a local inference endpoint. Use standard prompt engineering: provide system role, user query, and optionally KV-cache context from RAG. The model's reasoning_content block is structured (XML tags); parse that for audit trails. For high-throughput batch work (e.g., daily contract review), queue requests and run 3+ parallel slots on DGX Spark. Caching KV state across related queries is supported but requires llama.cpp v2.0+; confirm version compatibility before deployment.
When it's not the right fit
- —You need real-time sub-100ms latency. At 240–320 tok/s (RTX PRO 6000), a 500-token response takes 1.5–2 seconds; streaming helps but reasoning work is inherently batch-oriented.
- —Vision or multimodal input is required. Ornith's multimodal vocab is preserved, but the vision tower is *not* included in the GGUF; you'd need separate vision encoding + fusion (not part of this model).
- —You lack Blackwell GPU access. MXFP4 and NVFP4 optimization are Blackwell-native (sm_120 / sm_121). On older NVIDIA GPUs or AMD, you lose 1.5–2× MTP speedup and must dequantize to Q8_0 (~40 GB VRAM), likely violating cost/latency SLA.
- —Your company cannot own model maintenance. Ornith-1.0 is frozen (no updates promised by s-batman/DeepReinforce). If critical bugs appear in llama.cpp quantization or the MTP graft, you are responsible for patching or rolling back.
Alternatives to consider
Qwen3.6-35B-A3B (unquantized)
Same upstream MoE architecture; vastly larger VRAM footprint (~70 GB FP16); no speculative decoding built-in. Use if you need to fine-tune on non-Blackwell hardware or require full model weights.
Gemma 4-31B (Google)
Smaller parameter count, longer context (as of 2024-latest), commercial support from Google Cloud. Trade-off: less agentic-coding training and no MTP optimization; roughly comparable reasoning quality.
DeepSeek V3 / V4 (quantized variants)
Large MoE model with strong reasoning; Chinese origin may raise compliance flags in some orgs. Quantization support varies; no native MTP heads. Consider if you operate in China or require Chinese technical support.
Related open models
FAQ
Can we actually keep proprietary code and data in-house when running Ornith?
Yes—if deployed on your own Blackwell GPU cluster (on-prem or private cloud instance). All inference happens locally; model weights, KV cache, and outputs never leave your network. You control data egress via firewall rules and network segmentation. Responsibility: secure the Blackwell cluster and llama.cpp runtime yourself.
Is Ornith-1.0 licensed for commercial AI products?
Yes, MIT license permits it. You may fine-tune, distribute, and build products around Ornith without royalties. Limits: credit the original DeepReinforce authors in your model card; derivative models must also be MIT. You cannot claim Ornith as your own creation.
How does the multi-token prediction head affect reasoning quality?
No quality loss: MTP is a speculative-decoding mechanism, not a separate reasoning layer. The trunk's reasoning_content block is unchanged. MTP simply accelerates verification—output distribution is identical to non-MTP inference. Acceptance rates: 75–93% depending on sampling temperature (measured on RTX PRO 6000).
What if we don't have NVIDIA Blackwell GPUs?
Ornith will still run on older NVIDIA GPUs (A100, H100) via llama.cpp, but you lose the MXFP4/NVFP4 optimization. You'd need to dequantize to Q8_0 (~40 GB VRAM), roughly doubling memory cost and reducing throughput. Not recommended unless you're piloting before Blackwell rollout. AMD Instinct MI355X can run MXFP4 (OCP Microscaling) but not NVFP4.
Ready to build a private AI operations system?
LLM.co helps you deploy Ornith-1.0 on your own GPU infrastructure, integrate it into ops workflows, and customize it for code review, compliance, and internal knowledge automation—all with zero data leaving your environment. Talk to us about your Blackwell deployment or custom fine-tuning roadmap.