Open-Weight LLM · Private & Custom AI
Jan-nano-128k
Compact 4B model with native 128k context for document-heavy research automation and long-form operational workflows that run entirely in your infrastructure.
Jan-Nano-128k is a 4B-parameter open-weight LLM from Menlo Research, purpose-built with a native 128k token context window for research and document-processing tasks. For ops teams, it's a lightweight base for private deployment—handling multi-document synthesis, policy automation, and knowledge-base search without leaving your network. Its small footprint and context depth make it practical for mid-market custom AI and autonomous agent workflows.
Model facts
Private deployment
Run Jan-nano-128k in your own environment
Runs on modest hardware (estimated 8–16 GB VRAM in fp16, ~4–8 GB quantized). Deployable via VLLM or llama.cpp with rope-scaling configuration. Data never leaves your environment—Apache 2.0 license, no usage restrictions, and no external API calls required. Companies can embed this in internal Kubernetes clusters, air-gapped networks, or edge servers. Model card includes working deployment examples (VLLM/llama-server commands provided); community support via GitHub and HuggingFace discussions.
Operational AI use cases
Internal Knowledge & Policy Document Automation
Use Jan-Nano-128k to ingest entire SOPs, compliance manuals, and internal wikis (128k context covers most docs in one pass). Deploy as a private chatbot or workflow agent—no HR/Legal content leaks to external APIs. Ops teams query policy, and the model reasons over the full document set to answer eligibility, approval workflow, and compliance questions in real-time.
Multi-Turn Support Ticket & Incident Triage
Feed the full conversation history plus relevant KB articles into the 128k window. Classify tickets, extract follow-up actions, and route to the right team—all in your private environment. The extended context means fewer context-window swaps and faster ticket resolution; no sensitive customer data touches third-party systems.
Long-Form Report & Contract Analysis
Finance and Legal teams process 50+ page reports, contracts, or filings in a single inference. Extract obligations, risk flags, and action items without summarization loss. Deploy as a document ingestion service; route outputs to CRM, Jira, or email for downstream action. 128k context eliminates chunking overhead for typical business documents.
Custom AI
As a base for custom AI
Strong fit as a base model for internal-only AI products. Small size and long context enable rapid fine-tuning on proprietary datasets (customer support dialogue, internal Q&A, ops procedures) without expensive compute. Menlo provides the base; you add domain data and deploy the resulting model as a private service. Ideal for building vertical AI apps (e.g., internal research assistant, HR co-pilot) that must remain on-premises.
In the operating system
Where it fits
Sits at the **agent/reasoning layer** in an LLM.co-style OS. Acts as the stateful conversation engine for autonomous workflows: receives multi-step instructions, maintains context over long documents or conversation turns, and outputs structured decisions (routing, extraction, classification). Its 128k window makes it suitable for **knowledge orchestration**—the model can reason over injected doc context plus live API data without external retrieval calls, reducing latency and architecture complexity.
Data control & security
Self-hosted deployment keeps all data—user queries, internal docs, conversation history—within your network boundary. No training data sent externally; no usage telemetry. This is an architectural advantage: you own compute, storage, and inference logs. Note: Apache 2.0 license does not provide security/compliance certifications; your ops team must handle encryption, RBAC, audit logging, and isolation independent of the model. Suitable for HIPAA/SOC2/FedRAMP contexts *if* your infrastructure layer handles those guarantees.
Hardware footprint
**Estimate (verify in your environment):** fp16 = ~8 GB VRAM; int8 quantization = ~4–5 GB; GGUF Q4_K_M ≈ 2–3 GB. Batch size 1, max tokens 128k: add ~1–2 GB per concurrent inference. Recommended for single-node deployment or small clusters. GPU-optional for dev; production should use NVIDIA/AMD GPUs for sub-second latency. CPU-only inference possible for batch/async workflows (~5–30 sec/inference depending on hardware).
Integration
Expose via standard LLM APIs (VLLM/llama-server provide OpenAI-compatible endpoints). Integrate with orchestration layers (LangChain, LlamaIndex, Crew AI) for multi-step workflows. Connect to knowledge sources (vector DBs, document stores, APIs) via prompt injection or tool-calling (model supports hermes tool-call parser). Supports safetensors format for fast loading; quantization (GGUF) available for lower-latency inference on edge hardware. Chat template baked into tokenizer; handle jinja issues per FAQ (LMStudio workaround documented).
When it's not the right fit
- —You need model reasoning/"thinking" steps over complex multi-hop problems—Jan-Nano is explicitly a non-thinking model; consider Qwen3 or DeepSeek for reasoning tasks.
- —Your ops workflow requires sub-100ms latency on every inference; 4B model on modest hardware will struggle under high concurrency without aggressive quantization or GPU clustering.
- —Compliance mandates external model audits or vendor SLAs; open-weight models place audit/ops burden on your team, not a vendor.
- —You need multilingual support beyond English—Jan-Nano is primarily English-optimized; check benchmarks for your target languages.
Alternatives to consider
Qwen 3 (32B or smaller variants)
Larger, more capable, but requires more VRAM; better for complex reasoning. Also open-weight and Apache 2.0. Choose if your ops tasks need stronger language understanding and you have headroom on hardware.
Llama 3.2 (1B–11B variants)
Smaller and very well-supported by community/deployment tools. Shorter context window (~8k native), so choose if your docs/workflows fit smaller windows and you prioritize speed over depth.
Mistral 7B or Mixtral 8x7B
Comparable size/performance tier, strong instruction-following. Mistral 7B has 32k context; Mixtral more powerful but larger. Both permissively licensed; good alternatives if you need broader model ecosystem adoption.
Related open models
FAQ
Can I run Jan-Nano-128k on-premises with no external calls?
Yes. Apache 2.0 license, no restrictions. Deploy via VLLM/llama.cpp in your data center or private cloud. All inference happens locally; model weights are public on HuggingFace, so you control updates and versioning. No phone-home telemetry or API dependency.
Is commercial use allowed?
Yes. Apache 2.0 is a permissive OSI license. You can use Jan-Nano-128k in commercial products, fine-tune it for your business, and redistribute (with license preservation). No royalties or vendor approval required.
How do I integrate this with my existing ops tools (CRM, Jira, Slack)?
Deploy via VLLM's OpenAI-compatible API endpoint. Use webhooks or scheduled jobs to send docs/queries to that endpoint, parse responses, and post results back to your tool. Example: Zapier → VLLM → CRM. Or embed the model in a custom agent framework (LangChain, Crew AI) that orchestrates tool calls.
What if the 128k context isn't enough?
Document chunking + retrieval (vector DB, BM25) can supplement. For a single inference, 128k is the native limit (rope-scaling is already applied in deployment). If you need true multi-document reasoning, query a vector DB to fetch top-K relevant chunks, inject them, then run inference.
Build Your Private Ops AI System
Jan-Nano-128k is a powerful foundation for custom AI automation that stays inside your network. LLM.co helps mid-market teams design and deploy private LLM workflows—from fine-tuning on your data to integrating with your existing ops stack. Let's build an AI OS tailored to your business.