LLM.co — Private AI & custom AI developmentCall +1 (206) 844-1326All inference local

KV Cache Optimization for Self-Hosted LLMs Under Load

How paged attention, prefix caching, and quantized KV cache cut GPU memory for self-hosted LLMs so one cluster can serve more concurrent enterprise users.

Samuel Edwards5 min read
GPU servers packed in a private data center rack.

Why does a self-hosted LLM slow to a crawl the moment concurrency climbs past a handful of users? The KV cache fills the HBM long before the weights do, and every new request competes for the same GPU bytes.

The fix is rarely a bigger card. It is a smaller per-request KV footprint. Four knobs do most of the work: how the cache is laid out in memory, how it is reused across requests, how it is quantized, and how the model was built in the first place.

These are the levers custom LLM deployment teams reach for first when an H100 or MI300X fleet starts rejecting requests under load.

Where the HBM Actually Goes Under Load

The KV cache stores key and value tensors for every token already processed, across every layer, for every active request. Its size grows linearly with layer count, with the number of key/value heads, with context length, and with batch size. At BF16, a long-context Llama-class 70B at 128K tokens burns tens of gigabytes per request just on cache, before the weights, before activations, before any safety margin.

The original vLLM paper measured that naive allocators waste 60-80% of reserved KV memory to internal fragmentation and conservative pre-reservation, which is why PagedAttention exists in the first place. If the cache is the bottleneck, every optimization that follows is an attack on the same enemy: bytes per token per concurrent request.

PagedAttention and Prefix Reuse

PagedAttention treats the KV cache like virtual memory. Fixed-size blocks map to physical GPU pages, and a request only allocates what it has actually produced. On production vLLM, this routinely doubles the number of in-flight requests a GPU can hold at a given context length versus contiguous allocation, and it is the reason vLLM became the default open-source engine.

Prefix caching takes the same idea further. When many requests share a system prompt, a long retrieved document, or a chain-of-thought scaffold, the prefill work can be computed once and the KV blocks reused. vLLM's automatic prefix caching accelerates the prefill phase but not decode, which matters for RAG and multi-tool agents where the prompt is huge and the generation is short.

SGLang goes further with RadixAttention, a radix tree of cached KV segments that survives across generation calls. The LMSYS team reported up to 5x throughput over vLLM, Guidance, and TGI on structured generation workloads that reuse large prompt fragments. For chat and agentic tool-use traffic, this is where the free lunch is.

Illustration of an H100 GPU with HBM memory stacks and abstract paged cache blocks.

Quantizing the Cache Itself

The second lever is dropping precision on the KV tensors themselves. FP8 halves memory versus BF16 with almost no quality cost on well-behaved models. INT4 cuts it in half again at a measurable perplexity penalty. On Hopper and newer, the hardware pathway is there; on Blackwell, NVFP4 pushes further on both cache and weights.

The engine matters more than most teams expect. SqueezeBits measured FP8 KV cache in TensorRT-LLM at up to 1.09x throughput in prefill-heavy runs and 1.45x in decode-heavy runs, while vLLM gained nothing because its default FlashAttention-2 backend could not consume an FP8 cache at the time of the test. Quantizing weights with AWQ or GPTQ is a separate decision from quantizing the cache, and the gains compound only when the backend supports both paths at once.

Architectural Shortcuts: GQA, MLA, and Sliding Window

The cheapest optimization is picking a model that was already designed to spend fewer bytes per token.

Grouped-Query Attention shares a small number of KV heads across many query heads. Llama 3 70B uses 64 query heads but only 8 KV heads, which shrinks the KV cache by 8x relative to vanilla multi-head attention at the same depth. Qwen 3, Mistral Small 3, and Gemma 3 all ship with GQA for the same reason, and the ratio of query heads to KV heads is worth checking before any model is standardized on.

Multi-Head Latent Attention takes a more aggressive route. DeepSeek compresses keys and values into a low-rank latent that is decompressed on the fly, and the V2 paper reports a 93.3% KV cache reduction versus its predecessor alongside a 5.76x boost in maximum generation throughput.

Sliding-window attention, used in Mistral and parts of Gemma, caps the cache at the window size rather than the full context. Fidelity on long documents suffers if the window is too tight for the task, but for chat and short-RAG workloads the memory math is dramatic.

Overflow Valves: CPU Offload and Speculative Decoding

When a workload still does not fit, two tactics push the ceiling higher without buying GPUs.

CPU offload spills cold KV blocks to host DRAM over NVLink or PCIe and pages them back when a request becomes active again. Latency rises on cold hits, aggregate throughput on bursty traffic rises more. vLLM, TensorRT-LLM, and SGLang all expose offload knobs; the real decision is whether the SLO can absorb the extra time-to-first-token.

Speculative decoding attacks the decode side. A small draft model proposes several tokens at once and the large model verifies them in parallel, which collapses several memory-bound decode steps into one. Picking between inference engines partly comes down to which draft frameworks (Eagle3, Medusa, n-gram) they support cleanly and which workloads they have been tuned for.

Peak output throughput gain from Eagle3 speculative decoding in vLLM on gpt-oss-120b
Peak output throughput gain from Eagle3 speculative decoding in vLLM on gpt-oss-120bShareGPT (decode-heavy): 27.2%; SWE-bench (code): 24%; MLPerf TP=2 (prefill): 16%; MLPerf TP=1 (prefill): 9.5%ShareGPT(decode-heavy)27.2%SWE-bench (code)24%MLPerf TP=2 (prefill)16%MLPerf TP=1 (prefill)9.5%
Decode-heavy chat workloads benefit most; prefill-heavy long-prompt workloads see smaller but still real gains. Source: Red Hat, 2026

The Trade-Off You Are Actually Making

Every lever here trades something real. FP8 and INT4 KV quantization trade a measurable quality delta for roughly half the memory. Prefix and radix caching trade cold-start latency and cache-hit sensitivity for enormous reuse wins on repeat traffic. Sliding windows and MLA trade long-document fidelity for throughput. CPU offload trades tail latency for capacity. Speculative decoding trades a second model's weights for faster decode.

The reason the bundle is usually worth it is simple: the alternative is more hardware. For a regulated enterprise running a private LLM appliance or a self-managed cluster, doubling throughput on the same chassis extends the usable life of the capital already in the rack and defers the next procurement cycle.

Pick the quality floor evals will accept, fix a target P95 TTFT, and work through these levers in order of risk: cache layout and prefix reuse first, then architecture choice, then cache quantization, then offload and speculation. Review the stack whenever a new model family lands; the cheapest win is often the next generation's GQA ratio.

// written by
Samuel Edwards

Throughout his extensive 15+ year journey as a digital marketer, Sam has left an indelible mark on both small businesses and Fortune 500 enterprises alike. His portfolio boasts collaborations with esteemed entities such as NASDAQ OMX, eBay, Duncan Hines, Drew Barrymore, Price Benowitz LLP, a prominent law firm based in Washington, DC, and the esteemed human rights organization Amnesty International. In his role as a technical SEO and digital marketing strategist, Sam takes the helm of all paid and organic operations teams, steering client SEO services, link building initiatives, and white label digital marketing partnerships to unparalleled success. An esteemed thought leader in the industry, Sam is a recurring speaker at the esteemed Search Marketing Expo conference series and has graced the TEDx stage with his insights. Today, he channels his expertise into direct collaboration with high-end clients spanning diverse verticals, where he meticulously crafts strategies to optimize on and off-site SEO ROI through the seamless integration of content marketing and link building.

Bringing AI in-house, the right way.

Talk through your private or on-prem LLM deployment with an expert who has shipped them in regulated environments.

// the briefing

Private AI, in your inbox.

Occasional, high-signal notes on enterprise LLM deployment, security, and model strategy. No spam.