Open-Weight LLM · Private & Custom AI
gpt-oss-120b
120B MoE model for private, reasoning-heavy operational AI—automating complex workflows, agent systems, and internal knowledge tasks within a single H100.
gpt-oss-120b is an Apache 2.0 open-weight model (117B params, 5.1B active via MoE) from OpenAI, designed for reasoning, agentic work, and fine-tuning on company infrastructure. It runs on a single 80GB GPU with MXFP4 quantization, making it viable for self-hosted deployments where data never leaves your environment. For ops teams, this means building custom AI systems that stay private while retaining the reasoning depth of a 120B model.
Model facts
Private deployment
Run gpt-oss-120b in your own environment
Self-host on a single NVIDIA H100 (80GB) or AMD MI300X; no external API calls required. Download weights directly, run via vLLM, Transformers, or PyTorch. Data processing, agent reasoning, and outputs remain within your VPC. Trade-off: you own infrastructure cost (~$35–50k capital + ops); benefit is complete data isolation and no API rate limits or vendor lock-in.
Operational AI use cases
Internal knowledge agent
Route internal documents, process specs, and compliance logs through gpt-oss-120b to answer employee questions. Fine-tune on company-specific jargon and data formats. Reasoning modes (low/medium/high) let you dial latency vs. accuracy per query—critical for support ticket automation.
Workflow automation & function calling
Native function-calling and structured outputs enable chaining to CRM, ERP, or finance systems. Example: generate expense reports, approve workflows, or draft contract amendments by reasoning over data pulled from your internal APIs. Model runs locally; no external API exposure.
Agentic process mining
Deploy as a reasoning backbone for process discovery: feed logs, process data, and compliance rules; model identifies inefficiencies and drafts operational changes. Chain-of-thought visibility helps audit why decisions were made—mandatory for regulated ops (finance, healthcare).
Custom AI
As a base for custom AI
Fully fine-tunable on H100. Build domain-specific AI products (vertical SaaS, internal tools) by adapting gpt-oss-120b weights to your task. Harmony response format is baked in—keep it consistent during fine-tuning. No usage limits or API licensing friction; ship at your own cadence.
In the operating system
Where it fits
Sits at the reasoning/agent layer of an AI OS. Use it as the core LLM for multi-step workflows, replacing API-dependent chains (Claude/GPT-4). Pair with a vector DB (embeddings layer) and function-calling framework (integration layer) to build agentic automation that operates on internal data without external dependencies.
Data control & security
Self-hosting is an architecture choice: all inference, reasoning, and outputs happen inside your VPC or data center. No data shipped to OpenAI or a third-party API. This is control, not inherent security—you still own operational/access security, encryption at rest, and network controls. Audit your deployment setup independently.
Hardware footprint
**Estimate:** 80GB VRAM (MXFP4 quantization, as trained). FP16 full precision ~240GB. Smaller quantizations (int8) may fit 40–60GB but require custom kernel work. Batch size 1–4 typical for reasoning tasks (longer output); larger batches push memory quickly.
Integration
OpenAI-compatible API via vLLM or Transformers Serve; drop-in replacement for systems expecting OpenAI schema. Function calling uses standard JSON schema. Requires harmony response format—apply via chat template (auto in Transformers pipeline) or manually with openai-harmony package. Batch inference via Triton or PyTorch reference implementations. Integrate via REST/gRPC to business apps (no native connector; build/use OSS adapters).
When it's not the right fit
- —Real-time or sub-second latency required—reasoning tasks and 120B model size make p50 latency 2–5s+ per query.
- —Running on consumer GPUs (RTX 4090, etc.)—designed for data center GPUs; consumer porting is possible but unsupported and slow.
- —Light inference-only use case—overhead of managing a private H100 or cluster is unjustified if latency/cost of a hosted API is acceptable.
- —Regulated compliance without extensive audit trails—no built-in logging or compliance framework; you must layer those yourself.
Alternatives to consider
Meta Llama 3.3 70B
Open, smaller, fits on 80GB. Good for general reasoning; less agentic depth. Simpler fine-tuning path but fewer tool-use features out of the box.
Databricks DBRX (Instruct, 132B)
Similar scale, also MoE. Comparable reasoning for ops tasks. Databricks ecosystem lock-in if you want managed tuning; no OpenAI-native tooling.
Mistral Large (123B)
Dense model, good reasoning, MIT license. Runs on similar hardware; less transparency on training data and evals. No native agentic framework; more DIY integration.
Related open models
FAQ
Can I run this entirely on-premises without any cloud or external API?
Yes. Download weights, host on your H100, deploy vLLM or Transformers Serve behind your VPC firewall. Data never leaves your network. You manage infra scaling; no API lock-in.
What's the commercial use story?
Apache 2.0 license permits commercial deployment, product embedding, and for-profit use. Build SaaS, internal tools, or resell AI features freely. No royalties to OpenAI; you own the model weights and inference pipeline.
Do I have to use the harmony response format?
Yes. Model was trained with harmony format; using it raw will degrade outputs. Apply automatically via Transformers chat template or manually with openai-harmony package. Non-negotiable for good results.
How much does it cost to run vs. an API?
Capital: H100 ~$35–50k + networking/infra. Ops: power, cooling, sysadmin ~$500–2k/month. At 100k+ inferences/month, private hosting often cheaper than API pricing; below that, hosted APIs may win on CapEx.
Build Private, Reasoning-Powered AI Systems
gpt-oss-120b is a fully open-weight model built for self-hosted deployment. LLM.co helps you architect private LLM stacks, fine-tune for your ops workflows, and integrate reasoning agents into internal systems—keeping all data under your control. Explore building with gpt-oss-120b on LLM.co.