When Self-Hosting an LLM Actually Beats the API on Cost

Most CIOs assume self-hosting a model is cheaper than paying OpenAI or Anthropic by the token. In practice, that is only true above a specific traffic floor, and the floor is higher than the vendor pitches imply. Below it, an API contract wins on cash and on operational simplicity. Above it, an H100 or L40S node pays for itself in months and keeps paying for years.

This piece puts a number on where that line sits. It uses current list pricing, published inference throughput, electricity at commercial rates, and a fully loaded ops engineer. The output is a formula, not a vibe, and a defensible figure a CIO can walk into a budget meeting with.

So what does the break-even actually look like on paper?

The Inputs That Decide the Answer

Four variables set the economics. Token volume per day. Input-to-output ratio. Average context length. Required latency. Change any of them and the answer moves.

On the API side, pricing is public and stable enough to anchor a model. GPT-4o lists at $2.50 per million input tokens and $10 per million output tokens with a 128K context window. Claude Sonnet 4.5 sits slightly higher at $3.00 input and $15.00 output per million tokens, with cached input at $0.30. A standard 4:1 input-to-output mix on GPT-4o blends to roughly $4 per million tokens.

On the self-hosted side, the comparable workload is Llama 3.3 70B at FP8 on a single H100 80GB. That configuration is the current benchmark target because the weights alone consume most of the card. Smaller workloads fit on L40S nodes at a fraction of the cost per hour but with less headroom for long context or batching.

Hardware pricing has stratified. The OCPI-H100 settled index prints H100 SXM at $2.79 per GPU-hour on transacted neocloud deals as of October 2026, which works out to roughly $2,009 per GPU-month. Hyperscaler list prices run several times higher, with a median of $4.20 across 16 providers and AWS at $6.88. Buying outright puts a single H100 PCIe at $25,000 to $33,000 and a full HGX node north of $350,000 before networking.

Throughput Sets the Denominator

Cost per token on your own hardware is a function of GPU-hour price divided by tokens served per hour. The second number is where most TCO models go wrong. Single-stream decode on Llama 3.3 70B FP8 is slow, around 120 tokens per second at concurrency 1. The same hardware pushes about 1,850 tokens per second at 50 concurrent requests, and roughly 2,400 at 100.

Aggregate throughput is what matters for the economics. A single H100 running vLLM at steady 50-request concurrency produces about 6.6 million output tokens per hour. At the OCPI settled $2.79 GPU-hour, that is roughly $0.42 per million output tokens, before amortizing anything else. Compare that to the $10 GPT-4o output rate and the gap is 24x on the raw hardware line.

The catch is utilization. A node that runs at 50-way concurrency only four hours a day averages a fraction of that throughput over 24 hours. The honest denominator for an enterprise workload is somewhere between 20 percent and 60 percent sustained utilization, which is why batching, caching, and off-hours jobs matter so much. The deployment choices that drive utilization are the subject of a longer post on deploying open-weight models.

The Fully Loaded Node, Not Just the Sticker

GPU-hour pricing is only the headline. A defensible TCO adds four more lines:

• Amortization. An owned 8xH100 HGX node at $350,000 depreciated over 36 months is roughly $9,700 per month, or $1,210 per GPU-month. Add chassis, networking, and spares and you are closer to $1,500.
• Electricity. An H100 SXM draws 700W under load. At 70 percent duty cycle and $0.12/kWh commercial power, a single card runs about $43 per month. An 8-GPU node with CPU, NICs, and cooling overhead lands near $600 to $800.
• Colocation or on-prem facility. Rack space, cross-connects, and power delivery for an HGX node runs $1,500 to $3,500 per month in most metros.
• Fully loaded ops engineer. One SRE at a $220,000 fully loaded cost, allocated 25 percent to the LLM platform, adds $55,000 per year or about $4,600 per month per cluster.

Add those lines and an owned 8xH100 node lands near $18,000 per month all-in. That is $2,250 per GPU-month, meaningfully above the $2,009 OCPI rental index but with no termination risk and no provider margin. The breakdown mirrors what shows up in the full private AI cost breakdown: hardware is not the biggest line once you count people.

Where the Lines Actually Cross

Put the two curves on the same axis. API cost scales linearly with tokens. Self-hosted cost is roughly flat up to the throughput ceiling of the node, then steps up as you add GPUs.

At the GPT-4o blended rate of $4 per million tokens and a self-hosted blended rate of $0.80 per million at 40 percent utilization, the crossover sits around 4.5 million tokens per day for a single owned H100 against its fully loaded $18,000 monthly cost. Rented at the OCPI index on a reserved contract, the crossover drops to roughly 2 million tokens per day, because the fixed base is lower.

Against Claude Sonnet 4.5 at $18 per million blended, the lines cross earlier, closer to 1 million tokens per day. For a reasoning-heavy workload that leans on GPT-6 Astra at $10 input and $50 output per million, the crossover can arrive below 500,000 tokens per day. The arithmetic on replacing API subscriptions gets more aggressive the more premium the model you were paying for.

The usable formula:

• Break-even tokens per day = (monthly all-in node cost) / (30 x blended API $ per million x 1e-6) tokens
• For GPT-4o-class pricing and an $18,000 node: ~150 million tokens per month, or ~5M per day
• For Claude Sonnet 4.5-class pricing: ~33 million per month, or ~1.1M per day

Latency and context length pull the line sideways. A workload that needs single-request speed above 60 tokens per second cannot run at 50-way concurrency, so your throughput denominator collapses and the break-even moves up. A workload with 32K average context takes KV-cache headroom off the GPU and reduces how many concurrent sequences fit, which has the same effect. The GPU sizing writeup and the 200-user cluster post go deeper on how context budget trades against concurrency.

Where L40S Changes the Math

Not every workload needs an H100. The L40S has a current secure on-demand floor of $0.97 per GPU-hour, roughly a third of the H100 SXM rate. For 8B- to 13B-parameter models, or for 70B at aggressive quantization with modest concurrency, an L40S node can hit the same break-even at a far lower monthly base.

A 4xL40S node, owned and fully loaded, lands near $7,000 per month. Against GPT-4o blended pricing, the break-even sits closer to 1.75 million tokens per day. That is a realistic floor for a 500-seat internal RAG assistant or a document-processing pipeline. The hardware trade-offs are covered in detail in the LLM appliance and appliance versus DIY cluster writeups.

The Costs That Do Not Show Up in the Formula

Three things keep the API cheaper than the math suggests, and two things keep the self-hosted model cheaper.

APIs win on bursty traffic. A workload that peaks at 20M tokens per day for three days a quarter and sits at 100K the rest of the time will never fill a GPU. APIs also win when the model you need is proprietary and has no open-weight equivalent at the required quality for your task.

Self-hosted wins on three lines the formula does not price. Data residency, which is binding for HIPAA, GLBA, and CMMC Level 2 shops. Token-cost volatility, where vendors have raised prices more than once since 2023. And the compounding cost of context: every retrieval chunk, every system prompt, every tool call pays the per-token rate forever on an API, while on your own hardware it is free once the GPU is paid for. The private agent runtime changes the agent calculus specifically, because agentic workflows push input tokens hard.

Compliance is not in the formula but it is in the budget. Teams running regulated workloads should read the compliance walkthrough before signing an API contract they will have to unwind.

A Defensible Number for Your Budget Meeting

The honest answer for most regulated enterprises looks like this. Below 1 million tokens per day of GPT-4o-class usage, stay on the API. Between 1 and 5 million tokens per day, run the math carefully with your real latency and context numbers. Above 5 million tokens per day, or above 1 million on a Claude-class or reasoning-model workload, an owned or reserved H100 or L40S node is the cheaper line and the compliance posture improves at the same time.

The formula is portable. Plug in your own blended API rate, your own node cost, and your own realistic utilization, and the crossover falls out. What the formula will not do is tell you whether the data you are sending to the API is a liability on its own. That is a separate conversation, and usually the one that moves the decision.