Open-Weight LLM · Private & Custom AI
gpt-oss-20b
A 20B MoE model optimized for low-latency private deployment and custom ops automation with full reasoning chain transparency.
gpt-oss-20b is OpenAI's smaller open-weight model (21.5B parameters, ~3.6B active via MoE) designed for self-hosted deployment on consumer/mid-tier hardware and specialized ops workflows. Built on the Harmony response format, it exposes chain-of-thought reasoning and supports fine-tuning, function calling, and agentic automation—ideal for companies wanting to run reasoning workloads entirely within their own infrastructure.
Model facts
Private deployment
Run gpt-oss-20b in your own environment
Runs in ~16GB VRAM (MXFP4 quantized), down to consumer GPUs (e.g., RTX 6000, L40S) via Ollama, LM Studio, or vLLM. Company controls all data: no logs to vendor APIs, no third-party processing. Deployment paths: Ollama for local dev/POC, vLLM for production inference servers, Transformers for integration into Python pipelines. Harmony format requirement is non-negotiable—raw outputs will malform without it.
Operational AI use cases
Support ticket automation & triage
Route and draft responses for high-volume support queues. Use low/medium reasoning for fast triage, full chain-of-thought for complex cases. Fine-tune on internal ticket patterns to match company tone and SLAs. All ticket data stays on-prem.
Operational process documentation & knowledge retrieval
Extract and auto-generate runbooks, troubleshooting guides, and SOPs from internal logs and chat histories. Deploy as a rag agent with function calling to fetch internal wikis, Jira, or Slack without leaving your network.
Financial & compliance report drafting
Ingest raw financial statements, audit logs, and regulatory templates; generate first-draft reports with visible reasoning for audit trails. Reasoning levels let you trade latency for rigor (low for routine, high for year-end).
Custom AI
As a base for custom AI
Strong base for domain-specific applications: fine-tune on proprietary customer interactions, internal process data, or vertical-specific corpora without data leaving your environment. Supports structured outputs and function calling, enabling custom workflows (e.g., a billing agent that queries your ledger directly). Reasoning chains enable debugging and trust-building for regulated use cases.
In the operating system
Where it fits
Ideal as the reasoning/agent core in LLM.co's ops layer: handles complex workflow decisions, generates justifications for automation, and powers agentic tools (browser, code execution, function calling). Smaller sibling to the 120B; trades raw capability for latency and hardware cost. Sits between lightweight embedding/classification models and larger reasoning models in the inference stack.
Data control & security
Self-hosting ensures data never transits third-party APIs or vendor infrastructure—critical for PII, financial records, or proprietary ops data. Inference happens in your VPC or on-prem hardware you control. No claim that the model itself is 'secure' or 'compliant'; security posture depends on your deployment architecture (network isolation, access controls, encryption at rest/transit).
Hardware footprint
**Estimate:** ~16GB VRAM (MXFP4 quantized), ~20GB (FP8), ~40GB (FP16). Tested on H100/MI300X; runs on A100, L40S, or RTX 6000 with quantization. CPU inference possible via Ollama but latency penalties substantial for production.
Integration
Exposes OpenAI-compatible API via vLLM or Transformers Serve (drop-in replacement for dev pipelines). Supports function calling (schema-driven tool invocation), web browsing, and Python code execution when run with the gpt-oss SDK. Chain-of-thought must be parsed or hidden from end-users—use wrapper logic to extract reasoning vs. output. Requires Harmony format in prompts; Transformers chat_template handles it automatically.
When it's not the right fit
- —Context length is unknown—check model card or run benchmarks before committing to doc-heavy RAG workflows.
- —Requires Harmony format; incompatible with standard instruction-tuning pipelines unless you reformulate prompts.
- —Very long-form reasoning at high levels may exceed latency budgets on smaller GPUs; 120B model recommended for complex multi-step tasks at scale.
- —No published evals vs. proprietary reasoning models; relative performance on your ops tasks (e.g., ticket classification, compliance reporting) requires validation.
Alternatives to consider
Meta Llama 3.1 70B
Larger, single-model (no MoE), broader instruction-tuning, lower latency per token on mid-tier hardware. Better for general ops use; worse for memory-constrained setups.
Qwen QwQ-32B
Similar reasoning focus, smaller footprint, native structured outputs. Competitive for ops workflows; licensing may differ (verify commercial terms).
Mistral Large (open-weight via MoE)
Similar MoE architecture, lower VRAM, proven production track record. Less emphasis on chain-of-thought; better for latency-critical ops.
Related open models
FAQ
Can I run this on a single mid-market server without a GPU?
CPU inference is possible via Ollama but slow for production workflows. GPU recommended (16GB VRAM minimum with quantization). For CPU-only, consider smaller models or accept ~1-5s latency per request.
Is gpt-oss-20b commercially usable for building a customer-facing product?
Yes. Apache 2.0 is permissive: no copyleft, no patent encumbrance, no attribution required in binary form. You can commercialize products built on or fine-tuned from this model. Review OpenAI's Terms of Service for any runtime restrictions.
What's the Harmony format and why do I need it?
Harmony is the structured format gpt-oss was trained on. Using standard chat templates without it degrades output quality. Transformers chat_template applies it automatically; vLLM and direct inference require manual application or the gpt-oss SDK.
Can I see the model's reasoning before it returns an answer?
Yes—the model exposes full chain-of-thought. Extract it from the output, use it for debugging or audit trails, but strip it before showing end-users (it's internal working, not user-facing content).
Build Private Ops AI with gpt-oss-20b
Deploy reasoning-driven automation in your VPC. LLM.co helps you integrate, fine-tune, and scale open-weight models for support, finance, compliance, and agentic workflows—keeping all data in-house.