NVIDIA GPU Requirements for Running Private LLMs

Most sizing conversations for on-prem LLMs start at the wrong end. A vendor quotes an eight-GPU H100 node, the CIO signs, and three months later the serving team discovers that the model fits but the KV cache does not, concurrency collapses at 8K context, and a fine-tuning job pushes the box into swap. The hardware was never wrong in the abstract. It was wrong for the workload nobody wrote down.

The math behind NVIDIA GPU requirements for private LLMs is not complicated. It is just rarely done before the purchase order. This piece walks through the arithmetic a buyer should be able to defend to a compliance committee: how VRAM is actually spent, how concurrent users and context length erode the budget, and which NVIDIA SKU in the current lineup fits which class of model. Treat it as a worksheet, not a leaderboard.

Start With the VRAM Budget, Not the GPU

A GPU holds four things during inference: model weights, the KV cache, activations, and framework overhead (CUDA graphs, allocator fragmentation, the serving runtime itself). Weights are the easy number. A dense model at FP16 consumes roughly two bytes per parameter, so a 70B model lands near 140 GB. Quantize to INT4 and the same checkpoint fits in a much smaller footprint: roughly 42 to 46 GB for a 70B at INT4, with the published AWQ build of Llama 3.1 70B needing about 35 GiB just to load the checkpoint before any cache is allocated.

The KV cache is where sizing goes wrong. It is a per-token, per-layer tensor that lives in VRAM for the entire life of a request. The standard formula is 2 × layers × KV heads × head dimension × bytes per element per token. For a modern grouped-query model like Llama 3.1 70B (80 layers, 8 KV heads, head dim 128, BF16), that is 320 KB per token. For an older multi-head architecture at the same scale, it is closer to 2.5 MB per token, an 8× difference that single-handedly determines how many users a box can hold.

Multiply out and the budget disappears quickly. One Llama 3 70B request at a 128K context window burns about 40 GiB of KV cache on its own. Serve four of those concurrently and the cache alone exceeds the weight footprint of the model. Activations and runtime overhead typically add another 5 to 10 percent on top. None of this shrinks when you quantize the weights, which is the point most sizing spreadsheets miss.

Concurrency and Context Collapse the Budget

Two levers set the serving envelope: maximum context length and maximum concurrent requests. Both multiply the KV cache linearly. A deployment advertised as "70B at 128K context" means something very different if the real workload is twenty concurrent 4K chat sessions versus two concurrent 128K document reviews.

Run the numbers against a single 80 GB H100 holding an INT4-quantized 70B. The weights take about 45 GB, leaving roughly 30 GB of headroom after runtime overhead. At BF16 KV with GQA-8, that headroom holds about 96K tokens of aggregate cache. You can slice that as one 96K-token user, twelve 8K-token users, or twenty-four 4K-token users. Pick one.

This is why a serving stack like vLLM matters as much as the GPU under it. PagedAttention breaks the cache into fixed-size blocks so fragmentation does not strand memory, and continuous batching lets short requests return GPU cycles to longer ones instead of blocking the queue. The hardware choice sets the ceiling; the runtime decides how close you get to it. Teams that skip this layer end up rediscovering the same failures in the first ninety days of production.

Matching Model Class to NVIDIA SKU

The current data-center lineup covers a wide span of memory and bandwidth. The useful question is not "which GPU is fastest" but "which SKU holds the model I actually plan to serve, at the context length my users will actually hit, for the concurrency my logs already show." A quick map:

• L4 (24 GB GDDR6). Suits 7B-class models at modest context, embedding workloads, and small reranker services. A reasonable choice for a RAG retriever tier, not for a primary chat model.
• L40S (48 GB GDDR6). The Ada Lovelace L40S ships with 48 GB at 864 GB/s, 350 W TDP, PCIe Gen4, and does not support NVLink or MIG. Comfortable for a quantized 13B or 34B at production context, or a 70B at INT4 with short context and low concurrency. Dense FP16 70B serving is not its job.
• A100 (40 or 80 GB HBM2e). Still common in existing fleets. The 80 GB variant handles a 70B at INT4 and is a workable training GPU at smaller scale. Replace, do not expand, if you are buying new.
• H100 (80 GB HBM3, 94 GB on the NVL). The H100 SXM5 reaches 3.35 TB/s of memory bandwidth with FP8 Transformer Engine support. A single H100 serves a 70B at INT4 with real concurrency; a pair with NVLink serves a 70B at FP16. The default serving GPU for most private deployments today.
• H200 (141 GB HBM3e). Same Hopper compute as H100 with 141 GB at 4.8 TB/s. The right pick when the workload is memory-bound: long context, 70B-plus inference at scale, or multi-model serving on one card.
• B200 (192 GB HBM3e). The Blackwell B200 ships with 192 GB at up to 8 TB/s, a second-generation Transformer Engine, and native FP4. Overkill for a single 70B; justified for 100B-plus dense models, long-context 400B MoE serving, or a consolidation play where one card replaces three.

On Hopper and Blackwell parts, Multi-Instance GPU can split one card into up to seven isolated instances, each with its own memory and compute slice. For regulated tenants that want hard partitioning between workloads on the same box, MIG is more defensible than process-level isolation. L40S does not offer it, which matters for anyone writing an isolation story into a private LLM RFP.

H100 vs L40S for LLM Serving: Where the Line Actually Is

The H100-versus-L40S question comes up on almost every build because the sticker prices sit roughly 2.5× apart. The honest answer is that they are not substitutes; they serve different workloads.

Pick L40S when the serving target is a quantized model in the 7B to 34B range, context is bounded at 8K to 32K, concurrency is in the low tens, and the deployment is PCIe-only with no NVLink fabric. The 48 GB envelope is generous for that class and the 350 W TDP fits more server chassis without rewiring the rack.

Pick H100 (or H200) when the model is 70B or larger, when context extends past 32K, when concurrency reaches the hundreds, or when you need tensor parallelism across NVLink to hold a dense model that will not fit on one card. The PagedAttention design helps on both, but it cannot manufacture bandwidth. At batch size one, the H200's bandwidth advantage over H100 translates almost directly into tokens per second.

Training and Fine-Tuning Shift the Math

Inference sizing is the easier case. Training and fine-tuning add optimizer states, gradients, and activation checkpoints that typically push the memory requirement to four to six times the inference footprint for full fine-tuning, or roughly 1.2 to 1.5× for LoRA-style parameter-efficient methods. A 70B full fine-tune is a multi-node job on H100 or H200; a 70B LoRA is a single-node job on an eight-GPU H100 box if the batch is small.

Most private deployments do not need full fine-tuning. The practical workflow is a strong open-weights base, retrieval for freshness, and LoRA for behavior. The trade-offs between those three approaches are covered in RAG vs fine-tuning vs long-context, which is worth reading before the hardware spec is frozen. The GPU you buy for inference should also carry the fine-tuning plan you intend to run on it, not a plan you might one day want.

A Defensible Worksheet Before the PO

Before signing the hardware order, write down six numbers for each workload the box will serve:

1. Model and parameter count, with the exact quantization you will deploy.
2. Weight memory at that precision.
3. Per-token KV cache size, computed from the model's config.json, not estimated.
4. Target maximum context length (the real one, from logs if you have them).
5. Target maximum concurrent requests at that context.
6. Overhead reserve: 10 to 15 percent for runtime, allocator, and CUDA graphs.

Sum them. If the total exceeds the VRAM of a single card, you need tensor parallelism and an NVLink fabric, not a bigger PCIe GPU. If it fits with less than 15 percent headroom, size up — production traffic is not polite about staying inside the median. The same worksheet distinguishes an appliance order from a self-built GPU cluster, and from a pre-integrated LLM appliance where the sizing is already done against a published workload.

What This Means for the Buyer

The GPU market rewards buyers who show up with specifics. "We need 70B capability" is not a specification; it is a wish. "We serve a 70B INT4 model at 16K context for up to 60 concurrent users with 15 percent headroom, on a serving stack that supports PagedAttention and continuous batching" is a specification, and it narrows the SKU list to two or three options with real reasons behind each. That is the posture a CIO can defend to a board, and the one that produces hardware that still fits the workload a year later.

Private AI is a sizing problem before it is a model problem. The arithmetic above is the floor. Teams working through a first custom deployment or evaluating a managed path should do the worksheet first, pick the SKU second, and treat the vendor's reference architecture as a starting point rather than an answer. The models will keep changing. The memory bill will not.