vLLM vs TensorRT-LLM vs SGLang for Production Inference

Most private LLM deployments default to whichever inference engine the integrator demoed on the sales call. That is a bad reason to pick a runtime that will carry every token your organization generates for the next three years. The three engines a regulated buyer will actually shortlist in 2026 are vLLM, NVIDIA TensorRT-LLM, and SGLang. All three are Apache 2.0, all three now ship continuous batching and paged KV, and the easy feature-gap pitch from 2024 no longer applies.

What differs is the shape of the workload each one serves best, the hardware surface each one supports, and the governance you inherit along with the code. This piece lays out the selection framework for CIOs and infrastructure leads running a private LLM inference stack inside their own perimeter, so the choice is made on evidence rather than vendor momentum.

The Three Engines in One Paragraph Each

vLLM is the default. PagedAttention introduced block-level KV memory management that eliminated the fragmentation older servers suffered under concurrent load, and the project has since absorbed structured output, speculative decoding, multi-LoRA and prefix caching. It has been a PyTorch Foundation project since May 2025, which matters for buyers who want neutral governance rather than a single-vendor steward. vLLM also runs on the widest hardware surface of the three, covering ROCm, Intel XPU, CPU, and Apple silicon alongside NVIDIA.

TensorRT-LLM is NVIDIA's compiled path. The engine is Apache 2.0 across the repository, with the C++ runtime and CUDA kernels in the public tree rather than held back as a stub. The headline tradeoff is a build step: the model is compiled ahead of time into a GPU-generation-specific engine, which costs setup time and locks you to NVIDIA silicon in exchange for the highest sustained throughput of the three at high concurrency.

SGLang, maintained under LMSYS, is built around RadixAttention, a radix-tree index over KV cache that shares prefixes across requests. On agentic and multi-turn workloads where every request repeats the same system prompt and tool definitions, this is a structural advantage no amount of PagedAttention tuning will replicate. On traffic where every prompt is unique, it is close to a wash.

Throughput and Latency on Real Hardware

The numbers that matter for sizing a cluster are tokens per second at sustained concurrency and p95 time to first token. On an H100 running a 70B FP8 model at 100 concurrent requests, Spheron's 2026 benchmark measured TensorRT-LLM at 2,780 tok/s, SGLang at 2,460 tok/s, and vLLM at 2,400 tok/s. TensorRT-LLM leads at every concurrency level once compiled, and the gap widens as load climbs.

Latency tells a different story. At 100 concurrent requests on the same hardware, TensorRT-LLM reports a p95 TTFT of 1,280 ms against vLLM's 1,450 ms. That 170 ms difference is visible to a human on an interactive chat, but it is small compared to what a well-tuned prefix cache can do. On prefix-heavy traffic, SGLang's RadixAttention delivered a 37% lower TTFT p50 and a 41% lower p95 than vLLM at 50 concurrent requests. On unique-prompt traffic, SGLang's advantage over vLLM narrowed to roughly 3.8%, within benchmark noise.

Two implications. First, if your workload is a RAG system or agent fleet where every call shares a long system prompt, SGLang will often beat a compiled TensorRT-LLM engine on realized latency, even though TensorRT-LLM is faster on paper. Second, cold start times are not comparable. vLLM and SGLang reach a first served request in about a minute; TensorRT-LLM's compile path takes roughly 28 minutes in the same test, which matters if the deployment pattern is immutable images rebuilt per model update.

Scheduler Features That Matter in Production

Three scheduler capabilities decide whether an engine is production-ready for a regulated enterprise: continuous batching, prefix caching, and constrained decoding. All three engines now implement all three. The implementations are not equivalent.

Continuous batching, originally called in-flight batching, is table stakes. Prefix caching is where SGLang's architecture pays off: RadixAttention is on by default, uses arbitrary prefix boundaries rather than flat prefixes, and exposes a cache hit rate at the /metrics endpoint under sglang:cache_hit_rate. The original SGLang paper reported a 52.4% hit rate for LLaVA-Next-34B and 74.1% for Vicuna-33B after a month in Chatbot Arena. vLLM's automatic prefix caching is now enabled by default in V1 with under a 1% penalty at a 0% hit rate, but still indexes at the block rather than request level.

For constrained decoding (JSON schema, regex, grammar), SGLang's XGrammar implementation runs the lowest overhead at batch sizes of eight or more. TensorRT-LLM uses the same grammar engine. vLLM's structured output path is more expensive at scale, which matters if a majority of your traffic is tool calls or structured extraction. If you are building private AI agents whose outputs are consumed by downstream systems, this difference compounds.

Hardware Compatibility and Operability On-Prem

For a buyer who already owns the hardware, this is often the dispositive dimension. TensorRT-LLM runs only on NVIDIA GPUs. If your GPU plan includes AMD MI300X, Intel Gaudi, or anything else, it is not a candidate. vLLM and SGLang both support non-NVIDIA accelerators, though real-world maturity varies and most published benchmarks still run on H100 or H200.

Operability on-prem means three things: Prometheus metrics, log destinations your SIEM can ingest, and a predictable upgrade path. All three engines expose Prometheus endpoints. SGLang's covers request throughput, KV-cache utilisation, RadixAttention hit rate, scheduler queue depth and preemption counts under the sglang: prefix. vLLM's metric surface is comparable. TensorRT-LLM typically fronts with NVIDIA Triton or Dynamo, which adds a layer but also adds enterprise support contracts if you want them. For an air-gapped deployment, the compile step is non-trivial: engines are tied to specific GPU generations, so a rack with mixed H100 and H200 nodes needs more than one artifact.

Licensing and Governance

All three engines are permissively licensed. The governance under each license is different. vLLM sits under the PyTorch Foundation. SGLang is maintained by LMSYS, now backed by a venture-funded company. TensorRT-LLM is NVIDIA's. For a regulated buyer, the question is not which license is cleanest, but which steward's long-run interests align with yours. A single-vendor steward will optimize for that vendor's silicon; a VC-backed startup will optimize for growth; a foundation will optimize for broad adoption. None of those is wrong, but they produce different roadmaps.

One footnote worth checking in your legal review: the TensorRT-LLM repository's license file bundles third-party code under MIT, BSD-3-Clause and an LTX-2 Community License scoped to a single visual-generation model directory. For a text-only chat or RAG deployment this does not touch you, but a counsel sign-off on the full LICENSE file is cheap insurance.

A Selection Framework You Can Defend

Pick TensorRT-LLM when the fleet is NVIDIA-only, the model set is stable enough that recompiling per update is not a weekly event, and sustained throughput at high concurrency is the binding constraint. Batch generation, offline summarization and large back-office processing fit this profile.

Pick SGLang when prefix overlap is above roughly 60%. Agent fleets with a fixed system prompt, RAG systems where retrieval context dominates, and multi-turn chat all qualify. Confirm the fit by reading the cache hit rate at /metrics during a pilot. If it stays below 20%, your workload is not what you thought it was, and vLLM is likely the safer default.

Pick vLLM when you want one runtime for everything, your hardware roadmap includes non-NVIDIA accelerators, or your team values a neutral foundation steward over peak throughput. For most organizations running their first private deployment, this is where the risk-adjusted answer lands. Teams planning this build should also read the companion piece on deploying an open-weight LLM and the field notes on what breaks in the first 90 days, because the engine choice interacts with everything downstream of it.

The honest answer is that most mature private deployments end up running two of these in production: vLLM as the default serving tier and SGLang for the agent and RAG paths where prefix caching earns its keep. TensorRT-LLM shows up when a specific throughput target justifies the compile discipline. The engine is a tool, not a strategy. The strategy is keeping the data inside your perimeter and the audit trail intact; the engine is what you tune once that is settled.