LLM.co · Private AI economicsLLM Hosting & API Cost Calculator

API token pricing vs. GPU hosting pricing

APIs charge for consumption: input tokens, output tokens and, increasingly, cached tokens and cache writes, each at its own rate. Dedicated infrastructure charges for reserved capacity: GPU-hours, whether or not a request arrives. One is a variable cost, the other a fixed cost, and that difference drives everything else.

• Variable (API): cost scales linearly with tokens. No idle cost, no capacity planning, no minimum. The provider absorbs the idle hours and charges you a margin for doing so.
• Fixed (dedicated or owned): cost is flat until you need another replica, then steps up. Every idle hour is paid for, so the effective price per token falls as utilization rises.

That is why APIs are usually cheaper at low or bursty volume and dedicated infrastructure can win at high, steady volume. The crossover depends on the model's API price, the GPU price, the model's throughput on that GPU, and how evenly your traffic fills the hours you pay for. Output tokens matter most: they typically cost four to eight times as much as input on the API, and they are also the expensive tokens to generate on your own GPUs, because decode runs one token at a time.

Hidden costs of LLM API consumption

Per-million-token prices look small. These are the multipliers that make the invoice large:

• Repeated system prompts. A 3,000-token system prompt sent with a 500-token question means six-sevenths of every request is overhead. Prompt caching cuts the price of that overhead, often to 10% or less of the input rate, but only when the prefix is byte-identical across requests.
• Long conversation histories. Chat resends the whole thread each turn, so input grows with every turn and total input grows roughly with the square of conversation length.
• Reasoning tokens. Thinking models generate hidden reasoning billed at the output rate. Anthropic and OpenAI both count these tokens as output. A short visible answer can carry thousands of billed reasoning tokens.
• Autonomous agent loops. One task can mean 10–50 model calls, each resending tools, instructions and accumulated state. Calculator 5 models this explicitly.
• RAG retrieval. Retrieved passages are input tokens. Five 800-token chunks per question add 4,000 input tokens to every call.
• Retries and tool calls. Malformed tool calls, timeouts and validation failures all re-bill. A 10% retry rate is a 10% surcharge.
• Multi-agent orchestration. Planner, worker and critic agents each read the shared context, so total tokens grow with the number of agents.
• Cache write charges. Some providers bill writing a prefix into the cache at a premium over normal input (Anthropic charges 1.25× for its 5-minute cache). Caching pays off only when reads outnumber writes.
• Pricing changes. Prices move in both directions; Google's Gemini 4 introductory prices, for example, are scheduled to rise on December 1, 2026. Budget against a range, not a point.
• Multimodal inputs. Images, audio and PDFs are converted to tokens at rates set by each provider, often far more tokens than the equivalent text.

How much does it cost to host an LLM?

The bill for self-hosting comes down to how many GPUs you need and for how many hours. Five things set that number: model size, the precision of its weights, how much context each request carries, how many requests run at once, and the throughput you need to hit. Price per GPU-hour then turns GPU-hours into dollars, and the spread between providers is wide: verified on-demand B300 prices in our registry run from $6.94 to $17.80 per GPU-hour for the same silicon.

Model size and quantization set the floor

Weights must sit in GPU memory. At 16-bit precision (BF16) each parameter takes two bytes; FP8 takes one; 4-bit formats such as MXFP4 and NVFP4 take roughly half a byte plus scaling factors. Meta's Llama 4 Maverick 17B-128E Instruct has 400B parameters: its BF16 checkpoint is 803 GB and the FP8 release roughly 417 GB. That halving is the difference between two 8-GPU H100 nodes and one. Quantization isn't free. Lower precision can cost accuracy, and the GPU only speeds it up if it has matching tensor cores: FP8 needs Hopper, Ada, Blackwell or MI300-class hardware, and native FP4 math needs Blackwell or the MI355X.

Context length and concurrency set the headroom

On top of the weights, every in-flight request holds a key-value (KV) cache that grows with its context. A replica serving 64 users with long prompts can need as much memory for KV cache as for weights. That is why the calculator separates a minimum footprint (weights plus runtime, about 15% headroom) from a production one (about 50% headroom for batching).

Mixture-of-experts: cheap to run, expensive to hold

Most frontier open-weight models are now mixture-of-experts (MoE). Each token is routed to a few experts, so only a fraction of the parameters do work per token. Kimi K3.1 Pro activates 104B of its 2,800B parameters per token, about 4%. But the router can pick any expert, so every expert must stay resident: its 1,561 GB checkpoint needs roughly 1,795 GB of GPU memory before any KV cache. MoE cuts compute per token, which helps throughput and cost per token. It does not cut memory. Under real batch sizes, nearly every expert gets touched on every decode step anyway, so memory bandwidth, not FLOPS, usually sets the speed.

Throughput turns a monthly bill into a cost per token

A $50,000-a-month cluster is cheap at 30 billion tokens and ruinous at 300 million. The calculator estimates throughput from memory bandwidth and compute, then applies your utilization: the share of paid hours that serve real traffic. Most enterprise deployments run well below 100%, because traffic follows the working day.

Why GPU memory is not the same as inference capacity

A model that fits in memory can still be unusable in production. Fitting is the entry ticket. Capacity is how many tokens per second you can serve at acceptable latency, and four other limits decide it:

Memory bandwidth limits decode

Generating each output token means reading the active weights from memory. Generating one token at a time is bandwidth-bound: an H100 SXM moves 3.35 TB/s and a B300 about 8 TB/s. Batching many requests amortizes each weight read across many tokens, which is why throughput rises with concurrency until compute or KV-cache memory runs out.

Prefill and decode behave differently

Prefill (reading the prompt) processes thousands of tokens in parallel and is compute-bound; it is fast per token. Decode (writing the answer) is sequential. A workload with five input tokens per output token is mostly decode time even though most tokens are input, so the calculator weighs both by your input-to-output ratio.

The KV cache competes with the weights

Each concurrent sequence holds keys and values for its whole context. Long contexts and many users can fill the memory left after the weights, capping concurrency and therefore throughput. Architectures that compress attention, such as multi-head latent attention, sparse attention and linear-attention hybrids, exist mainly to shrink this cost.

Parallelism needs the right interconnect

A model too big for one GPU is split. Tensor parallelism divides each layer across GPUs and exchanges activations every layer, so it needs NVLink or Infinity Fabric. Over PCIe it slows sharply beyond two to four GPUs. Expert parallelism places different MoE experts on different GPUs and routes tokens between them. Pipeline parallelism places layers on different nodes and tolerates slower links. A model spanning two nodes needs InfiniBand or RoCE between them and an engine that supports multi-node serving, such as vLLM or SGLang. The hosting calculator flags configurations that have enough total memory but the wrong topology.

Hidden costs of self-hosting large language models

The GPU line item is the visible part. A credible total cost of ownership also carries:

• GPU acquisition and lead time. Datacenter GPUs have no public list price. Quotes vary with volume, and delivery can take months.
• Electricity and cooling. An 8-GPU Blackwell server draws on the order of 10–15 kW. Multiply by power usage effectiveness (PUE) for cooling and conversion losses; the industry average is about 1.56.
• Colocation. Wholesale colocation is priced per kW per month, and high-density AI racks often need liquid cooling that older halls can't provide.
• Network infrastructure. Multi-node inference needs 400–800 Gb/s fabric, switches and optics: a capital cost that is easy to forget.
• Model storage. Trillion-parameter checkpoints run to terabytes, and you will keep several versions.
• Security. Hardening, secrets, network segmentation, vulnerability management and audits for the serving stack.
• Engineering and monitoring. Someone owns upgrades, driver and engine versions, autoscaling, dashboards, alerting and on-call.
• Redundancy. One replica is a single point of failure. N+1 doubles a one-replica bill.
• Hardware replacement. GPUs fail, and a four-to-five-year life means budgeting the next purchase before this one is paid off.
• Idle capacity. Traffic that peaks at 10 a.m. and idles overnight leaves owned GPUs dark for much of the day. Utilization is the single biggest lever on cost per token.
• Model upgrades. A new model may need different memory, a different precision or a new engine release. Plan for re-validation, not just a weights swap.

When is self-hosting an LLM cheaper than using an API?

Self-hosting wins when a fixed monthly bill is spread over enough tokens to beat the API's per-token price. That depends on:

• Monthly token volume. Below the financial break-even, the API is cheaper by definition.
• Sustained GPU utilization. Break-even assumes the tokens fit in the hours you pay for. At 30% utilization the effective GPU price is more than three times the list rate.
• Input-to-output ratio. Input-heavy workloads such as RAG and classification favor self-hosting, because prefill is cheap on your own GPUs while the API still charges per input token. Output-heavy generation narrows the gap.
• Quantization. A 4-bit model needs half the GPUs of an FP8 one, if quality holds.
• Prompt caching. API cache discounts lower the API side of the comparison, often a lot. Engines like vLLM and SGLang also reuse prefixes on your own hardware.
• Batch processing. Offline jobs can run at full utilization overnight, and many APIs discount batch traffic by about 50%.
• Model routing. Sending easy requests to a small model changes both sides of the comparison.
• Engineering overhead and depreciation. Staff time and hardware write-downs are fixed costs that raise the break-even point.
• Peak versus sustained demand. Capacity must cover the peak, but cost per token is earned on the average.
• Actual achievable throughput. Use a measured figure from your own load test whenever you can; estimates can be off by a factor of two in either direction.

That is why there is no universal break-even threshold. Every model has a different API price, every GPU a different rental rate, and every model–GPU pair a different throughput. A break-even quoted without those three numbers isn't meaningful. The calculator above shows two figures. The financial break-even is the volume where a fixed bill equals the API bill. The feasibilitycheck asks whether one replica can actually serve that volume. Sometimes it can't, and adding replicas just moves the break-even further out.

Kimi K3.1 Pro: API vs. dedicated B300s

Moonshot AI's Kimi K3.1 Pro is open-weight and also sold by API at $2.00 input, $0.20 cached input and $10.00 output per million tokens (verified on 2026-11-14, source). Its 1,561 GB MXFP4 checkpoint needs about 1,795 GB of GPU memory, so one 8× B300 node (2,304 GB) holds it with thin KV headroom and 8× B200 (1,440 GB) does not: B200 deployments need 16 GPUs. Its license is MIT-style, but Model-as-a-Service businesses above US$20M in revenue need a separate agreement.

API cost at a 5:1 input-to-output ratio, 50% of input cached

Each million input tokens, with its 200,000 output tokens, costs $3.10 on the API.

Dedicated capacity that can hold the model

At these prices the cheapest verified option, RunPod at $40,530 a month, reaches a theoretical cost break-even at about 13.1B input and 2.61B output tokens per month. That is a financial break-even, not proof the node can process that much. Whether one 8-GPU node sustains it depends on measured throughput at your context lengths and concurrency. Calculator 4 estimates this, and it shows that at 60% utilization a full replica can cost more per token than the API. Self-hosting Kimi pays when you can keep the GPUs busy, or when data-handling requirements rule the API out.

These prices replace an early draft table: several of its figures (Verda, Nebius and AWS B300 rates) no longer match the providers' own pages. Hyperstack's pricing page lists a B300 rate while its banner still describes B300 on-demand as upcoming, so confirm availability.

Private LLMs vs. public APIs for enterprise security

Cost is rarely the only reason enterprises self-host. These are the security questions that actually decide it:

• Data sovereignty. Where is the prompt processed and stored, and under which jurisdiction? Many APIs now offer regional processing (often at a premium); self-hosting puts it wherever you put the hardware.
• Confidentiality. Major API providers contractually commit not to train on business API traffic, but data still crosses your boundary. For privileged legal material, unreleased financials or classified work, crossing the boundary may itself be the problem.
• Contractual obligations. Client contracts, NDAs and data processing agreements may restrict sub-processors. Check whether your AI provider is an approved one.
• Retention policies. API providers keep logs for abuse monitoring unless you qualify for zero-data-retention terms. On your own stack, retention is your setting.
• Regulation. HIPAA, GLBA, GDPR, ITAR, CJIS and sector rules set obligations on how data is handled. They do not ban third-party processing: a signed BAA or DPA with a compliant provider can satisfy them.
• Vendor dependency. Models are deprecated, prices change and terms get rewritten. Open weights you host can be pinned and kept.

Two cautions apply in both directions. Self-hosting does not make a system compliant by itself: access control, encryption, audit logging, retention and incident response still have to be built and evidenced. And a third-party API is not inherently insecure: a well-governed API with the right contract can be the more secure choice than an under-maintained private cluster. See AI governance for the controls we put around either.

Hybrid AI infrastructure: the best of both models

Most enterprises end up hybrid. Neither pure option meets every need at a sensible price, and a hybrid estate typically combines:

• Privately hosted open-weight models for high-volume, repetitive or sensitive work: extraction, classification, summarization, internal search.
• Premium third-party APIs for the hardest reasoning on non-sensitive data, where a frontier model's quality is worth its price.
• Intelligent model routing that sends each request to the cheapest model able to handle it, with escalation when confidence is low.
• Local embeddings so documents are vectorized inside your boundary, and private RAG infrastructure so retrieval never ships your corpus to a third party.
• A centralized AI gateway that every application calls, so routing, rate limits, cost attribution and redaction live in one place.
• Access controls and audit logging tied to your identity provider, recording who asked what, which model answered and what data it saw.
• Data classification policies the gateway enforces: regulated or confidential requests are pinned to private models; public or low-risk ones may go out.

In practice that means contract review, patient summaries and internal search run on a private model, while a marketing team's brainstorming and a developer's open-source coding questions go to a frontier API. The break-even calculator's hybrid line models exactly this: a share of tokens routed to the API and the rest on owned hardware. Compare it with our hybrid deployment work.