Open-Weight LLM · Private & Custom AI
qmd-query-expansion-1.7B-gguf
A 1.7B specialized query-expansion engine for hybrid search pipelines—fine-tuned to transform raw search queries into structured lexical, vector, and hypothetical-document formats that operational teams can embed in private knowledge systems.
QMD Query Expansion is a small, MIT-licensed language model trained to take user search queries and expand them into three complementary formats (BM25 keywords, vector embeddings, HyDE passages) for QMD's hybrid retrieval system. For ops and AI teams building private knowledge automation, this is a production-ready tool: small enough to run locally, purpose-built for search quality, and fully open to customize or retrain on proprietary data.
Model facts
Private deployment
Run qmd-query-expansion-1.7B-gguf in your own environment
Deploy via Ollama or llama.cpp using the provided GGUF quantization. The model card includes a complete pipeline: merge SFT + GRPO adapters, quantize to Q4_K_M (~1.7B parameters ≈ 1–2 GB VRAM), then serve locally. No API calls, no data leaving your infrastructure. A company can run this on a single CPU machine or integrate it into an existing vector-database + BM25 backend without touching external services.
Operational AI use cases
Internal Documentation Search & Knowledge Base Automation
Ops teams maintain sprawling internal wikis, runbooks, and FAQs. Embed QMD Query Expansion before your vector DB to expand employee queries like 'reset password' into lexical variants ('password reset procedure'), natural phrases ('how to reset my account password'), and HyDE passages ('Password resets are initiated via the admin console'). Result: higher recall without tuning BM25 weights manually or hiring a search engineer.
Support Ticket Routing & Self-Service Automation
Customer support receives tickets with short, often vague queries ('login broken', 'billing issue'). Pipe incoming ticket summaries through QMD Query Expansion to generate multiple search angles, then match against a private knowledge base of resolved tickets and solutions. Use the expanded queries to auto-classify, suggest self-service articles, or route to the right team—all in-house, with no external API dependency.
Intelligent Document Retrieval for Compliance & Audit Workflows
Finance, legal, and compliance teams need to find relevant policies, contracts, and audit logs from massive document stores. QMD's three-pronged expansion (keyword, natural language, HyDE) ensures regulators' queries like 'vendor risk assessment' map to both policy keywords and semantic meaning. Self-hosted execution keeps sensitive docs from touching external models; teams can fine-tune on proprietary legal language and retain full audit trails.
Custom AI
As a base for custom AI
This model is a launchpad for custom operational AI: fine-tune it on your proprietary queries and expansions using the published SFT + GRPO pipeline (included in the model card). A company with domain-specific search terminology (e.g., medical coding, manufacturing procedures, financial instruments) can rerun the training scripts on ~1,000 labeled examples of (query → expansions) to create a custom model that understands their vocabulary. The small size (1.7B) means fast iteration; the open license means no licensing friction.
In the operating system
Where it fits
In an LLM.co-style AI operating system, QMD Query Expansion sits at the **knowledge layer**: it enriches user queries before they hit vector stores, BM25 engines, or retrieval-augmented generation (RAG) pipelines. It bridges the **agent/workflow layer** (where users or bots submit questions) and the **knowledge store** (vector DB + document corpus). It's not a reasoning model; it's a precision tool that makes downstream retrieval smarter without adding latency or external dependencies.
Data control & security
By running QMD locally (via Ollama or llama.cpp), your raw search queries never leave your infrastructure—they remain in your database and logs. This is an *architecture* advantage, not a claim about the model itself: any third-party model can be self-hosted. The benefit for ops: you control the entire pipeline, audit query patterns, and keep proprietary search behavior private. Compliance teams can verify no queries are logged externally. For regulated industries (healthcare, finance, law), this architecture is often a requirement.
Hardware footprint
**Estimate (varies by quantization):** Q4_K_M GGUF ~1.0–1.5 GB VRAM; FP16 full precision ~3.5–4 GB. Inference latency ~100–500ms per query on a single GPU (A10G, RTX 3090) or ~1–3s on a modern CPU. For comparison, a 7B model in the same quantization is ~3–4 GB. This 1.7B fits on edge devices, embedded inference servers, or a shared VM without dedicated GPU.
Integration
Integrate via a local HTTP endpoint (Ollama exposes `/api/generate`; llama.cpp has a compatible REST API). In your search pipeline: (1) receive raw query from user/agent, (2) POST to QMD endpoint with prompt `/no_think Expand this search query: {user_query}`, (3) parse the response (lex:, vec:, hyde: lines), (4) route lex: to BM25, vec: to vector embedding + search, hyde: to a second embedding pass. Adapts to existing Elasticsearch, Pinecone, or local Milvus backends. The model is deterministic (no sampling needed); set temperature=0 for production.
When it's not the right fit
- —You need general-purpose language understanding (reasoning, code generation, creative writing). QMD is a specialist: it only knows how to expand search queries in its training distribution.
- —Your queries fall outside English or technical/operational domains. The model is trained on ~1,000 English examples; domain drift (e.g., poetry, non-Latin scripts) will degrade expansion quality.
- —You need real-time multi-language support. QMD is monolingual; the model card shows only English tags. Retraining on multilingual data is possible but not included.
- —Your search backend doesn't support three-pronged retrieval (BM25 + vector + HyDE). If you only have a vector DB with no keyword search, the lex: output is wasted; a simpler paraphrase model might suffice.
Alternatives to consider
Llama 2 / Llama 3 (7B–8B, fine-tuned for RAG)
Larger, general-purpose base models with broader instruction-following; easier to customize for other tasks, but 4–5× the memory footprint and no built-in query-expansion training pipeline. Better if you want a single model to handle search + other operations.
Qwen2.5 or Qwen3 (base, 4B–7B)
QMD's parent model. Run the parent directly if you want more flexibility (multi-task fine-tuning, longer context). QMD is already optimized for query expansion; Qwen base is a blank slate.
Jina AI Reranker (open-source versions)
Reranking-focused, different objective: takes a query + candidate docs and scores relevance. Complements QMD (expansion → retrieval → reranking). Lighter weight for ranking, but doesn't solve query expansion itself.
Related open models
FAQ
Can I run QMD entirely on-premises without any cloud calls?
Yes. Download the GGUF file, spin up Ollama or llama.cpp locally, expose an HTTP endpoint, and route your queries through it. The model card includes full GGUF conversion and Ollama setup instructions. No external API keys, no data egress.
Is the MIT license permissive for commercial products?
Yes. MIT permits commercial use, modification, and distribution, including in closed-source products. You can sell a product that uses QMD; you must include the MIT license notice. No patent indemnity or trademark rights; consult legal if trademark matters.
What if my domain (e.g., medical, legal) needs custom expansions?
The model card provides the complete SFT + GRPO training pipeline. Collect ~1,000 examples of (short_query → your_ideal_expansions) in your domain, run `train.py sft --config configs/sft.yaml`, then GRPO, then GGUF conversion. Total cost ~$2–3 in cloud GPU time; total time ~2 hours. You'll own a fine-tuned model that understands your terminology.
How do I integrate this into my existing search stack (Elasticsearch, Pinecone, etc.)?
Add a query-expansion step upstream of your search: intercept the raw query, send it to QMD (local HTTP), parse the lex:/vec:/hyde: lines, then run lex: terms through your keyword search, vec: through your vector backend, and optionally use hyde: as a synthetic query for a second embedding pass. Most integration takes <100 lines of middleware code.
Build Smarter Search Into Your Private AI System
QMD Query Expansion is a fully open, fine-tunable specialist model ready to embed in your operational AI stack. Own your search intelligence, keep your data private, and customize it to your domain. Start with LLM.co's private deployment framework to integrate QMD with your vector stores, knowledge bases, and workflows—no external dependencies, full audit control.