Open-Weight LLM · Private & Custom AI
gpt-oss-120b
120B open-weight MoE for private reasoning agents and custom ops automation—runs on a single 80GB GPU with full chain-of-thought visibility.
gpt-oss-120b is OpenAI's 120B-parameter mixture-of-experts model (5.1B active), post-trained with MXFP4 quantization to fit a single H100/MI300X. Built for reasoning-heavy tasks, agentic workflows, and function calling, it ships with unfettered Apache 2.0 licensing. An ops team would deploy it privately to automate support escalation, document analysis, and workflow orchestration without data leaving their infrastructure.
Model facts
Private deployment
Run gpt-oss-120b in your own environment
Self-host via vLLM (0.10.1+) on NVIDIA CUDA or AMD ROCm; Red Hat provides validated container images (RHOAI 2.25, RHAIIS 3.2.2) and Kubernetes/OpenShift deployment templates. Model weights (~120GB unquantized, reduced via MXFP4) pull from HuggingFace or Red Hat's OCI registry. Architecture keeps inference, fine-tuning, and chain-of-thought logs entirely within your environment—no telemetry or external API calls. Requires Harmony response format compliance; vLLM handles this automatically.
Operational AI use cases
Support escalation and triage automation
Route incoming tickets by reasoning through support history, FAQs, and SLAs. Full chain-of-thought output (not shown to users) aids debugging and compliance audits. Function calling enables auto-ticketing, reassignment, and knowledge base lookups—reducing manual triage by 40–60%.
Internal documentation and knowledge synthesis
Index PDFs, runbooks, and policy docs; use the model as a private agent to answer internal queries (HR, compliance, IT) with source attribution. Reasoning effort can be tuned (low for FAQ, high for policy interpretation) to balance latency vs. quality.
Workflow approval and decision automation
Feed structured business data (expense reports, purchase orders, contract terms) and let the model reason through approval rules. Output structured decisions that integrate with your approval systems via API; audit trails capture full reasoning.
Custom AI
As a base for custom AI
Fine-tune gpt-oss-120b on your proprietary ops workflows, domain vocabulary, and reasoning patterns (model card explicitly permits parameter tuning). Train on in-house support transcripts, internal decision logs, or compliance case studies; deploy the custom variant alongside base for A/B testing. Harmony format is baked in, so no additional prompt engineering overhead. Use case: a healthcare ops team could fine-tune on anonymized patient-flow data to optimize scheduling and resource allocation.
In the operating system
Where it fits
Place gpt-oss-120b at the *reasoning and decision layer* of an AI operating system. Feed it structured inputs from a workflow orchestrator (Zapier, n8n, custom state machine); use its chain-of-thought output to populate audit logs and trigger downstream actions (approvals, notifications, data writes). Pair with a smaller model (gpt-oss-20b) for simpler tasks to manage latency and cost.
Data control & security
Self-hosting eliminates data transit to third-party APIs—reasoning outputs, chain-of-thought logs, and fine-tuning data remain in your VPC/Kubernetes cluster. No model telemetry or usage reporting home to OpenAI. Does *not* guarantee compliance (HIPAA, SOC 2, etc.); your ops team must enforce encryption, access controls, and audit logging around the deployment. Quantization (MXFP4) does not materially alter model safety—review your usage policy and use outputs responsibly.
Hardware footprint
Estimate (unquantized full precision): ~240GB VRAM. With MXFP4 quantization applied: ~80GB VRAM (fits single NVIDIA H100 or AMD MI300X). Smaller gpt-oss-20b: ~16GB with quantization. Context length and batch size will vary; max_model_len 32768 validated. Exact footprint depends on serving framework (vLLM tensor parallelism, batch settings).
Integration
vLLM exposes OpenAI-compatible `/v1/chat/completions` and `/v1/completions` endpoints; plug into Langchain, LlamaIndex, or curl. Function-calling and structured outputs are native; define JSON schemas inline. Red Hat templates include kservectl bindings for Kubernetes; extract outputs to Postgres, S3, or webhook endpoints for downstream approval systems. Transformers library also supported but vLLM recommended for production inference latency.
When it's not the right fit
- —You need sub-100ms latency for real-time user-facing chat—120B reasoning incurs ~2–10s per completion depending on reasoning effort and hardware.
- —Your ops workflows require deterministic, rule-based decisions with zero ambiguity; language models hallucinate, and chain-of-thought alone does not eliminate risk.
- —You cannot allocate 80GB+ GPU memory or lack access to NVIDIA/AMD enterprise GPUs; edge or mobile deployment is infeasible.
- —Your organization forbids open-source or non-proprietary models for regulatory reasons (some orgs only accept closed, certified solutions).
Alternatives to consider
Meta Llama 3.3 70B
Smaller, cheaper, fits 2× H100s, permissive Llama 2.1 license; less reasoning depth, but strong for ops workflows with lower latency expectations.
Mistral Large 123B
Similar parameter count, Apache 2.0, multi-language support; less OpenAI-native integration, but strong on multilingual ops (support teams in 10+ languages).
Anthropic Claude (self-hosted via Antml/Claude.ai API)
Proprietary but strong reasoning; requires cloud API (not private), so different architecture trade-off; consider if you accept external API calls for reasoning quality gain.
FAQ
Can I fine-tune gpt-oss-120b on proprietary support data and keep it fully private?
Yes. Download the weights, run training on your private GPU cluster with your data, and serve the fine-tuned variant from your VPC. Apache 2.0 permits commercial fine-tuning. No requirement to share weights or disclose training data.
What does 'Harmony response format' mean? Will my existing LLM integrations break?
Harmony is a structured prompt/response protocol that gpt-oss was trained on; vLLM and Transformers auto-apply it via the chat template. If you use the library's chat/completions endpoints, you're fine. If you use raw model.generate(), you'll need to apply the format manually or use the openai-harmony package.
Is Apache 2.0 safe for a company building internal ops tools on top of this model?
Yes. Apache 2.0 is permissive (OSI-approved), no copyleft, no patent risk. You can modify, fine-tune, and deploy commercially without attribution or source-code disclosure. Review your legal team's stance on open-source; most enterprises approve Apache 2.0.
How does MXFP4 quantization affect reasoning quality? Do I need to worry about degradation?
OpenAI post-trained the model with MXFP4 already applied; all evals used the same quantization, so published benchmarks reflect real performance. You don't get a 'full precision' option—this is the model. Reasoning quality is intact for ops tasks; if you need raw accuracy, test on your own workflows first.
Build private reasoning into your ops stack.
gpt-oss-120b runs entirely on your hardware, with full chain-of-thought visibility and fine-tuning flexibility. Let LLM.co help you integrate it into your workflows—data stays in your VPC, reasoning stays under your control. Contact us to architect a private AI system.