Open-Weight LLM · Private & Custom AI
Mamba2-primed-HQwen3-8B-Instruct
A hybrid attention-SSM 8B model for ops teams deploying long-context AI at 2× inference speed in their own infrastructure, trading some reasoning quality for throughput and memory efficiency.
Mamba2-primed-HQwen3-8B-Instruct merges 50% Attention layers with 50% Mamba-2 state-space layers, keeping context windows up to 128K while halving KV cache overhead. For ops use—document processing, support triage, long-context automation—it runs faster and cheaper on private hardware than the base Qwen3-8B, with instruction-tuning built in for direct deployment.
Model facts
Private deployment
Run Mamba2-primed-HQwen3-8B-Instruct in your own environment
Self-hosting this model keeps your long-context inference—customer transcripts, policy documents, internal knowledge bases—within your environment. At ~8B parameters in bfloat16, it fits on a single modern GPU (24–40GB VRAM estimate); the fixed-size SSM state means memory doesn't balloon with context length, allowing higher concurrency. You control the data, the inference, and model updates. Deployment requires standard Hugging Face/vLLM tooling; Amazon's Hybrid Model Factory repo supplies priming code if you later want to customize the architecture.
Operational AI use cases
Long-Context Support Ticket Auto-Triage
Ingest entire ticket histories (8K–128K tokens) with attachments and prior interactions in a single pass. The hybrid architecture's lower memory footprint lets you batch multiple tickets concurrently; inference at 128K context is ~2.3× faster than pure Attention, reducing latency for real-time triage. Feed output into rule-based workflows or a secondary classifier to route to the right team.
Policy / Compliance Document Q&A Agent
Load full employee handbooks, SOCs, or regulatory docs (100K+ tokens) into context once, then answer questions without re-ingesting. The 128K native support eliminates chunking overhead; the 50% SSM hybrid keeps inference cost flat relative to document length. Private deployment ensures sensitive HR/legal text never leaves your network.
Internal Knowledge Base Search & Summarization
Index and retrieve your internal wikis, past RFPs, or ops runbooks. Hybrid model's speed advantage scales with context—at 64K–128K tokens, you get 2× throughput vs. pure Transformer. Batch multiple user queries on the same large context window (e.g., "summarize Q3 incident reports") to amortize inference cost and cut end-to-end latency.
Custom AI
As a base for custom AI
Strong foundation for custom ops AI. Instruction-tuned by default, so minimal SFT needed for domain tasks. The hybrid architecture is transparent to your training pipeline—fine-tune it like any Transformer, or leverage Amazon's Hybrid Model Factory to re-prime it with your own data. Use it as a backbone for RAG, workflow agents, or structured extraction on internal documents. Smaller than 70B models but larger than 3B, positioning it in the sweet spot for ops automation without excessive compute.
In the operating system
Where it fits
Lives in the core reasoning / generation layer of an ops AI stack. Deploy as: (1) a stateless API endpoint for knowledge retrieval and summarization; (2) a long-context agent backbone for multi-step automation (ticket triage → resolution steps); (3) a low-latency batch processor for bulk document work. Below it: your knowledge base and vector store. Above it: your workflow orchestration, approval gates, and business system integrations (CRM, ERP, ticketing). The private deployment means this layer stays in-house.
Data control & security
Running this model on your own infrastructure means documents, transcripts, and internal text never transit through third-party inference APIs. You control storage, access logs, and retention. This is an architecture choice—the model itself has no built-in encryption or audit trail. Compliance teams should pair self-hosting with standard data governance (encryption at rest, network isolation, audit logging) to meet regulatory requirements. No formal security certification claimed; treat it as you would any open-weight model you deploy.
Hardware footprint
**Estimate** ~16–18 GB VRAM in bfloat16 precision (standard deployment). At full 128K context, peak KV cache is lower than Qwen3-8B Transformer (~4–6 GB for the SSM layers' fixed state, ~10–12 GB for Attention layers' KV). Fully quantized (int8, int4) can drop to 8–10 GB. Multi-GPU inference (TP=8) on modern instances: 8× H200 shows 2,825 tokens/s at 128K; smaller setups scale proportionally. Batch size × context length drives final memory; SSM layers' constant state means batching is memory-efficient.
Integration
Standard transformers/vLLM integration. Accepts text input and streaming output; works with OpenAI-compatible endpoints (vLLM) or Hugging Face generate() APIs. Pair with LangChain for prompt templating, memory, and tool calling. Wire output into your ops stack: call your CRM/ticketing API from model responses, feed summaries into your workflow DAG, or log decisions for audit. Context length (128K) lets you concatenate multiple business objects (ticket + conversation + docs) into a single inference request without fragmenting logic across API calls.
When it's not the right fit
- —Complex reasoning or math at scale: benchmarks show ~3-point gap vs. Qwen3-8B (Long) on short-context reasoning (GSM8K, MATH). Better for ops/doc tasks than reasoning-heavy chains.
- —Your team needs chain-of-thought thinking tokens: this model is instruction-tuned, not a thinking model; it doesn't natively produce reasoning traces.
- —You need the absolute highest quality on short contexts: on Tulu3-dev short-context tasks, Mamba2 variant averages 68.02 vs. Qwen3-8B (Long) at 71.21. If your workflow is mostly <8K tokens, pure Attention may be worth the latency trade-off.
- —Unfamiliar hybrid architectures present risk: Mamba-2 SSM layers are less mature than Attention. If you need guaranteed support/ecosystem depth, pure Transformer models have broader tooling.
Alternatives to consider
Qwen3-8B (pure Transformer)
Higher quality on reasoning/math, no architectural novelty risk, larger HF ecosystem. Use if your ops use is short-context (≤32K) or reasoning-heavy, and you can afford the slower inference at long contexts.
Llama 3.1 8B
Well-established, strong on short-context ops tasks (coding, instruction-following), single vendor. No long-context optimization; at 128K you'd rely on external scaling (scaling laws). Lighter community adoption than Qwen but solid for general-purpose ops.
Mistral 7B Instruct
Smaller footprint (7B vs. 8B), focused on instruction-following, widely deployed. Native context ~32K; no long-context efficiency gains like Mamba2. Good if your docs stay <32K and you want maximum memory headroom.
FAQ
Can I run this on my own hardware without sending data to an API?
Yes. Download the model from HuggingFace, run it with vLLM or Ollama on your own GPU or on-prem servers. All inference stays in your environment. You handle deployment, scaling, and updates.
What's the commercial / business use license?
Apache 2.0, ungated. You can use it commercially—no registration, no restrictions on output, no royalties. You're responsible for compliance with your own data and use-case regulations.
How much faster is this than the base Qwen3-8B on long documents?
At 128K context, ~2.3× faster throughput (2,825 vs. 1,227 tokens/s on H200 cluster). At 64K, ~2× faster. The advantage grows with context length because SSM layers' cost is constant while Attention KV cache shrinks the memory budget for batching in the base model.
Will I lose quality if I switch from Qwen3-8B to this Mamba2 hybrid?
On long-context tasks (8K–128K), quality drops ~3 points on average. On short-context ops benchmarks (Tulu3-dev), ~3 points down as well. If your ops workflows are mostly <32K tokens and don't demand high reasoning, the speed gain (2×) likely outweighs the quality trade. For pure reasoning or coding, stick with pure Attention.
Build a Private, Long-Context Ops AI System
Mamba2-primed-HQwen3-8B is a blueprint for fast, cost-effective private AI. Whether you're automating document workflows, agent-based triage, or knowledge search in your own environment, LLM.co helps you deploy, fine-tune, and scale hybrid or custom models. Let's design a system that keeps your data private and your ops faster.