LLM.co — Private AI & custom AI developmentCall +1 (206) 844-1326All inference local

Sizing KV Cache for Concurrent Users on Private GPUs

How to budget KV cache memory for concurrent users on a self-hosted LLM, using PagedAttention, FP8 quantization, prefix sharing, and chunked prefill.

Eric Lamanna6 min read
Rows of GPU server racks in a private, on-premises data center.

KV cache sizing is routinely quoted as a model-level overhead, a single number alongside the weights. In reality the formula batch_size × seq_len × 2 × n_layers × n_kv_heads × head_dim scales per-token and per-user, and Llama 3.1 70B at 128K context books roughly 42 GB per request before any weights load.

PagedAttention block accounting, FP8 or INT4 KV quantization, SGLang chunked prefill, and cross-request prefix sharing each move that per-request bill by a measurable factor. On an 8×H100 or 2×H200 node, their combination decides whether a procurement line item supports twenty concurrent analysts or two hundred.

An earlier piece on KV cache optimization introduced the mechanics; the numbers below put them against specific hardware budgets so a CIO can defend a concurrency target without hand-waving.

The Formula Multiplies Across Every User and Token

Expanding the arithmetic with Llama 3.1 70B parameters — 80 layers, 8 KV heads under grouped-query attention, 128-dim heads, 2 bytes per element in BF16 — yields 2 × 80 × 8 × 128 × 2 = 327,680 bytes per token, or roughly 0.33 MB. At 128K context that reaches about 42 GB per request. Eight concurrent analysts at 32K each book 86 GB in aggregate, already exceeding the quantized weights on an 80 GB card.

FlashAttention-3 raises attention compute to 740 TFLOPS on H100, three-quarters of the theoretical ceiling, but changes nothing about this arithmetic. The K and V tensors still have to live in HBM for every active token of every live request. GPU sizing for a private LLM deployment begins at this per-user memory footprint, not at parameter count or FLOPS headroom.

KV cache size by context length, across four open-weight models (FP16, GB)
Qwen3 8B
1.1 GB → 18 GB▲ 16.9 GB
Qwen3 14B
1.3 GB → 20 GB▲ 18.8 GB
Qwen3 32B
2 GB → 32 GB▲ 30 GB
Llama 3.3 70B
2.5 GB → 40 GB▲ 37.5 GB
Each row is the same context-length ramp (8K, 32K, 128K) on one shared GB scale, so the per-model slopes can be compared at a glance. Source: Cloud GPU, 2026

Grouped-Query Attention Rewrote the Denominator

The n_kv_heads term is where architectural choices show up on the invoice. Llama 3.1 70B ships with 64 query heads but only 8 KV heads, an 8× reduction over classical Multi-Head Attention and the single largest intervention in the formula. Llama 3.1 8B uses a 32:8 ratio for a 4× cut. Multi-Head Latent Attention pushes further, compressing K and V into a shared low-rank latent and dropping long-context DeepSeek-class models by roughly two orders of magnitude at the same context window.

The procurement implication: a model's KV budget is not a function of parameter count. It is a function of layers × KV heads × head dim. A 70B with GQA often costs less per concurrent user than a 13B designed before GQA was standard, which reorders the intuitive "bigger model, bigger node" rule.

KV heads per layer: MHA baseline vs GQA as shipped
KV heads per layer: MHA baseline vs GQA as shippedLlama 3 70B: 64; Llama 3.1 8B: 32MHA KV headsGQA KV headsLlama 3 70B648Llama 3.1 8B328
GQA replaces most KV heads with shared ones: an 8x cut for 70B and 4x for 8B, directly dividing the per-user KV cache bill. Source: Morph LLM, 2025

PagedAttention Trades Fragmentation for Block Flexibility

Classical contiguous KV allocation wastes 60-80% of reserved HBM to fragmentation because each sequence reserves its maximum possible length up front. vLLM's PagedAttention treats the cache as fixed 16-token blocks allocated on demand, which holds internal fragmentation under 4% and eliminates external fragmentation entirely.

The effect on concurrent capacity is direct. An 8×H100 node holds 640 GB of HBM. Under contiguous allocation at 32K context on Llama 3.1 70B, reservation overhead would cap useful KV cache at a fraction of what the formula predicts, because every active request would hold its full 32K slot whether or not it had generated that many tokens. PagedAttention packs roughly four times as many live requests into the same budget by returning unused blocks to the pool between generation steps. Block size itself is tuned: smaller blocks reduce fragmentation further but inflate the indirection table, and 16 tokens is the practical balance that landed in the reference implementation.

Close-up of an eight-GPU HGX baseboard with high-bandwidth memory stacks visible.

FP8 and INT4 Shrink the Cache Itself

Weight quantization is standard practice; KV cache quantization still lags behind it in production deployments. FP8 halves per-token bytes, replacing BF16's 2 bytes with 1. INT4 quarters them. For Llama 3.1 70B at 32K context with eight concurrent users, BF16 bookings reach 85.9 GB, FP8 lands at 42.9 GB, and NVFP4 comes in at 21.5 GB. The quality cost for FP8 KV on retrieval-heavy benchmarks is typically under half a percentage point, a cheap trade for a 2× bump in concurrency headroom.

The gains compound with batching. Continuous batching plus PagedAttention achieves up to 23× higher throughput than naive static batching, with continuous batching alone accounting for roughly an 8× improvement over the Hugging Face baseline. Layering FP8 KV on top of that stack roughly doubles concurrent capacity again at identical latency targets, provided the workload is not latency-bound at the smaller critical batch sizes where decode stalls dominate.

Prefix Sharing Stretches the Bill Across Users

When analysts share a system prompt, a retrieval preamble, or a tool manifest, their prefixes are identical for thousands of tokens. SGLang's RadixAttention and vLLM's prefix caching detect those shared runs and point multiple requests at the same KV blocks instead of recomputing them. In published community measurements, a 4,000-token shared system prompt reached a 99.6% hit rate and 7.5× throughput on a single L40S. RadixAttention has reported up to 6.4× throughput on prefix-heavy workloads against prior systems.

The fragility is architectural rather than tunable. Inserting a timestamp or request ID into the top of the prompt invalidates every downstream block, dropping hit rate from 89% to 3% in the same reproduction. Teams running the four-way bakeoff between vLLM, TensorRT-LLM, and SGLang sometimes forget that engine choice is downstream of the prompt template. Chunked prefill then stops long shared prefixes from starving short decode requests: SGLang defaults to 8K-token chunks, and mixed-load benchmarks show 40-70% TTFT reduction compared to blocking prefills on the same hardware.

Per-user KV footprint at 32K context, eight concurrent requests
Per-user KV footprint at 32K context, eight concurrent requestsBF16: 85.9 GB; FP8: 42.9 GB; NVFP4: 21.5 GBBF1685.9 GB · 57%FP842.9 GB · 29%NVFP421.5 GB · 14%
Same Llama 3.1 70B workload at 32K with eight users, sized by area. Treemap is used here for magnitude rather than strict parts of a whole: the three tiers are mutually exclusive precision choices for the same slot. Source: Spheron Network, 2026

What a 2×H200 or 8×H100 Node Actually Holds

An 8×H100 node carries 640 GB of HBM. Allocate 140 GB for Llama 3.1 70B weights in BF16, reserve 60 GB for framework overhead and activations, and about 440 GB of KV budget remains. At 32K context with FP8 KV (5.24 GB per request packed under PagedAttention) that supports roughly 80 concurrent analysts before TTFT degrades materially. INT4 KV roughly doubles that headroom to 160, at the cost of a validation pass on the specific workloads.

NVIDIA H200 SXM headline figures
141
HBM3e memory
1.76x H100 SXM capacity
4.8
Memory bandwidth (TB/s)
1.4x H100 SXM bandwidth
700
Max TDP (W)
Configurable, SXM form factor
Three independent headline figures for the H200 SXM. Capacity drives concurrent users; bandwidth drives decode latency. Source: NVIDIA, 2026

A 2×H200 node carries 282 GB, a smaller footprint with higher per-GPU bandwidth. After weights and overhead, roughly 100-120 GB of practical KV budget remains, which is 20-25 concurrent users at 32K FP8. The H200's extra bandwidth shows up more in decode latency than in raw capacity. Prefix sharing can then multiply those user counts again when the template is stable, which is exactly where production deployments earn back their hardware spend.

The question that opened this piece — how many concurrent analysts a given GPU budget actually supports — resolves cleanly once context length, KV precision, attention scheme, and prefix cache hit rate are each treated as 2-10× knobs rather than defaults. Teams pricing an on-prem cluster without applying all four tend to be off by an order of magnitude in the quote. A managed LLM appliance or a custom LLM deployment can back the arithmetic into a specific SLA tied to named workloads, which is the number procurement is signing against.

// written by
Eric Lamanna
Director of Business Development

Eric Lamanna is a Digital Sales Manager with a strong passion for software and website development, AI, automation, and cybersecurity. With a background in multimedia design and years of hands-on experience in tech-driven sales, Eric thrives at the intersection of innovation and strategy—helping businesses grow through smart, scalable solutions. He specializes in streamlining workflows, improving digital security, and guiding clients through the fast-changing landscape of technology. Known for building strong, lasting relationships, Eric is committed to delivering results that make a meaningful difference. He holds a degree in multimedia design from Olympic College and lives in Denver, Colorado, with his wife and children.

Bringing AI in-house, the right way.

Talk through your private or on-prem LLM deployment with an expert who has shipped them in regulated environments.

// the briefing

Private AI, in your inbox.

Occasional, high-signal notes on enterprise LLM deployment, security, and model strategy. No spam.