Sizing KV Cache for Concurrent Users on Private GPUs
How to budget KV cache memory for concurrent users on a self-hosted LLM, using PagedAttention, FP8 quantization, prefix sharing, and chunked prefill.

KV cache sizing is routinely quoted as a model-level overhead, a single number alongside the weights. In reality the formula batch_size × seq_len × 2 × n_layers × n_kv_heads × head_dim scales per-token and per-user, and Llama 3.1 70B at 128K context books roughly 42 GB per request before any weights load.
PagedAttention block accounting, FP8 or INT4 KV quantization, SGLang chunked prefill, and cross-request prefix sharing each move that per-request bill by a measurable factor. On an 8×H100 or 2×H200 node, their combination decides whether a procurement line item supports twenty concurrent analysts or two hundred.
An earlier piece on KV cache optimization introduced the mechanics; the numbers below put them against specific hardware budgets so a CIO can defend a concurrency target without hand-waving.
The Formula Multiplies Across Every User and Token
Expanding the arithmetic with Llama 3.1 70B parameters — 80 layers, 8 KV heads under grouped-query attention, 128-dim heads, 2 bytes per element in BF16 — yields 2 × 80 × 8 × 128 × 2 = 327,680 bytes per token, or roughly 0.33 MB. At 128K context that reaches about 42 GB per request. Eight concurrent analysts at 32K each book 86 GB in aggregate, already exceeding the quantized weights on an 80 GB card.
FlashAttention-3 raises attention compute to 740 TFLOPS on H100, three-quarters of the theoretical ceiling, but changes nothing about this arithmetic. The K and V tensors still have to live in HBM for every active token of every live request. GPU sizing for a private LLM deployment begins at this per-user memory footprint, not at parameter count or FLOPS headroom.
Grouped-Query Attention Rewrote the Denominator
The n_kv_heads term is where architectural choices show up on the invoice. Llama 3.1 70B ships with 64 query heads but only 8 KV heads, an 8× reduction over classical Multi-Head Attention and the single largest intervention in the formula. Llama 3.1 8B uses a 32:8 ratio for a 4× cut. Multi-Head Latent Attention pushes further, compressing K and V into a shared low-rank latent and dropping long-context DeepSeek-class models by roughly two orders of magnitude at the same context window.
The procurement implication: a model's KV budget is not a function of parameter count. It is a function of layers × KV heads × head dim. A 70B with GQA often costs less per concurrent user than a 13B designed before GQA was standard, which reorders the intuitive "bigger model, bigger node" rule.
PagedAttention Trades Fragmentation for Block Flexibility
Classical contiguous KV allocation wastes 60-80% of reserved HBM to fragmentation because each sequence reserves its maximum possible length up front. vLLM's PagedAttention treats the cache as fixed 16-token blocks allocated on demand, which holds internal fragmentation under 4% and eliminates external fragmentation entirely.
The effect on concurrent capacity is direct. An 8×H100 node holds 640 GB of HBM. Under contiguous allocation at 32K context on Llama 3.1 70B, reservation overhead would cap useful KV cache at a fraction of what the formula predicts, because every active request would hold its full 32K slot whether or not it had generated that many tokens. PagedAttention packs roughly four times as many live requests into the same budget by returning unused blocks to the pool between generation steps. Block size itself is tuned: smaller blocks reduce fragmentation further but inflate the indirection table, and 16 tokens is the practical balance that landed in the reference implementation.

FP8 and INT4 Shrink the Cache Itself
Weight quantization is standard practice; KV cache quantization still lags behind it in production deployments. FP8 halves per-token bytes, replacing BF16's 2 bytes with 1. INT4 quarters them. For Llama 3.1 70B at 32K context with eight concurrent users, BF16 bookings reach 85.9 GB, FP8 lands at 42.9 GB, and NVFP4 comes in at 21.5 GB. The quality cost for FP8 KV on retrieval-heavy benchmarks is typically under half a percentage point, a cheap trade for a 2× bump in concurrency headroom.
The gains compound with batching. Continuous batching plus PagedAttention achieves up to 23× higher throughput than naive static batching, with continuous batching alone accounting for roughly an 8× improvement over the Hugging Face baseline. Layering FP8 KV on top of that stack roughly doubles concurrent capacity again at identical latency targets, provided the workload is not latency-bound at the smaller critical batch sizes where decode stalls dominate.
Prefix Sharing Stretches the Bill Across Users
When analysts share a system prompt, a retrieval preamble, or a tool manifest, their prefixes are identical for thousands of tokens. SGLang's RadixAttention and vLLM's prefix caching detect those shared runs and point multiple requests at the same KV blocks instead of recomputing them. In published community measurements, a 4,000-token shared system prompt reached a 99.6% hit rate and 7.5× throughput on a single L40S. RadixAttention has reported up to 6.4× throughput on prefix-heavy workloads against prior systems.
The fragility is architectural rather than tunable. Inserting a timestamp or request ID into the top of the prompt invalidates every downstream block, dropping hit rate from 89% to 3% in the same reproduction. Teams running the four-way bakeoff between vLLM, TensorRT-LLM, and SGLang sometimes forget that engine choice is downstream of the prompt template. Chunked prefill then stops long shared prefixes from starving short decode requests: SGLang defaults to 8K-token chunks, and mixed-load benchmarks show 40-70% TTFT reduction compared to blocking prefills on the same hardware.
What a 2×H200 or 8×H100 Node Actually Holds
An 8×H100 node carries 640 GB of HBM. Allocate 140 GB for Llama 3.1 70B weights in BF16, reserve 60 GB for framework overhead and activations, and about 440 GB of KV budget remains. At 32K context with FP8 KV (5.24 GB per request packed under PagedAttention) that supports roughly 80 concurrent analysts before TTFT degrades materially. INT4 KV roughly doubles that headroom to 160, at the cost of a validation pass on the specific workloads.
A 2×H200 node carries 282 GB, a smaller footprint with higher per-GPU bandwidth. After weights and overhead, roughly 100-120 GB of practical KV budget remains, which is 20-25 concurrent users at 32K FP8. The H200's extra bandwidth shows up more in decode latency than in raw capacity. Prefix sharing can then multiply those user counts again when the template is stable, which is exactly where production deployments earn back their hardware spend.
The question that opened this piece — how many concurrent analysts a given GPU budget actually supports — resolves cleanly once context length, KV precision, attention scheme, and prefix cache hit rate are each treated as 2-10× knobs rather than defaults. Teams pricing an on-prem cluster without applying all four tend to be off by an order of magnitude in the quote. A managed LLM appliance or a custom LLM deployment can back the arithmetic into a specific SLA tied to named workloads, which is the number procurement is signing against.
Eric Lamanna is a Digital Sales Manager with a strong passion for software and website development, AI, automation, and cybersecurity. With a background in multimedia design and years of hands-on experience in tech-driven sales, Eric thrives at the intersection of innovation and strategy—helping businesses grow through smart, scalable solutions. He specializes in streamlining workflows, improving digital security, and guiding clients through the fast-changing landscape of technology. Known for building strong, lasting relationships, Eric is committed to delivering results that make a meaningful difference. He holds a degree in multimedia design from Olympic College and lives in Denver, Colorado, with his wife and children.

KV Cache Optimization for Self-Hosted LLMs Under Load

What a Private GPU Cluster for 200 Users Actually Costs

Docker, GPUs, and Distributed LLMs: A DevOps Guide

NVIDIA GPU Requirements for Running Private LLMs

Why DeepSeek’s Data Storage Policy Should Concern Privacy-Conscious Users

LLM Appliance vs Self-Built GPU Cluster for Private AI
Bringing AI in-house, the right way.
Talk through your private or on-prem LLM deployment with an expert who has shipped them in regulated environments.
Private AI, in your inbox.
Occasional, high-signal notes on enterprise LLM deployment, security, and model strategy. No spam.











