Skip to content

Guide library

Comparison · Hosting & inference

vLLM vs SGLang vs TensorRT-LLM: Which to Use

The three leading open-source LLM inference engines compared on hardware, prefix caching, disaggregated serving and release churn, with setup commands and a clear pick for each situation.

Metarix Editorial · Nov 14, 2026 · 11 min read

Documentation-based research

NVIDIA TensorRT-LLM illustration: a green model cube feeding optimized inference to a workstation

vLLM, SGLang and TensorRT-LLM are the three open-source engines most teams pick between when they serve open-weight models on their own GPUs. All three are free, Apache 2.0 licensed and expose an OpenAI-compatible API, so the real differences are hardware support, how they reuse cached prompts, and how much operational churn you can absorb.

This comparison covers what each engine is good at as of Nov 14, 2026, a feature and hardware table, the commands to start each one, what published performance numbers can and can't tell you, and a decision flow for choosing.

Quick answer

Start with vLLM if you want the safe default: the largest contributor base, the longest list of supported models and accelerators, and the most material when something breaks. Try SGLang when your traffic reuses long prefixes, such as agents, multi-turn chat or RAG with a fixed system prompt, or when you need its hierarchical KV cache. Benchmark TensorRT-LLM when your fleet is all NVIDIA and squeezing the most tokens per GPU justifies tracking frequent release candidates. Whatever you choose, test it on your own model, prompts and SLOs before committing.

Scroll the table horizontally on smaller screens.

vLLM vs SGLang vs TensorRT-LLM at a glance
SpecvLLMSGLangTensorRT-LLM
MaintainervLLM community (started at UC Berkeley's Sky Computing Lab)LMSYS (non-profit), sgl-projectNVIDIA
LicenseApache 2.0Apache 2.0Apache 2.0 (per README)
Latest stable releasev0.32.0 (Oct 18, 2026); v0.33.0 in release candidatesv0.5.23 (Nov 2, 2026)v1.2.1 (Apr 20, 2026); v1.4.0rc3 (Nov 6, 2026)
Start a servervllm serve <model>sglang serve <model>trtllm-serve <model>
HardwareNVIDIA, AMD, Intel GPUs, CPUs; TPU, Gaudi, Ascend and others via pluginsNVIDIA, AMD, Intel, Google TPU, Ascend, Apple Silicon, Moore ThreadsNVIDIA GPUs only
Signature featurePagedAttention KV memory managementRadixAttention prefix reuse, HiCacheNVIDIA-tuned kernels and FP8/FP4 recipes
Best fitDefault choice, mixed hardware, widest model coverageAgentic, multi-turn and prefix-heavy trafficAll-NVIDIA fleets chasing peak efficiency
Sources: each project's GitHub README, documentation and releases page, accessed Nov 14, 2026. Release dates are GitHub release dates (UTC).

What an inference engine actually does

An inference engine turns a model checkpoint into a server that can handle many requests at once. The hard part isn't running the model; it's scheduling. The engine batches requests continuously, splits work into prefill (reading the prompt) and decode (generating tokens), and manages the KV cache, the per-request memory that grows with every token and usually decides how many users fit on a GPU.

That is why the metrics that matter are time to first token, inter-token latency and goodput rather than a single tokens-per-second figure. If those terms are new, start with our explainer on LLM latency and throughput.

Diagram of where vLLM, SGLang and TensorRT-LLM sit between your app, routing layers and the hardware they run on
All three engines sit behind the same OpenAI-style API; they differ in caching, tooling and supported hardware. Graphic: Metarix, Nov 14, 2026.

vLLM: the broad default

vLLM began at UC Berkeley around PagedAttention, a way to store the KV cache in fixed-size blocks so memory isn't wasted on fragmentation. Its README now lists continuous batching, chunked prefill, prefix caching, CUDA and HIP graphs, torch.compile, speculative decoding (n-gram, suffix, EAGLE and DFlash), and disaggregated prefill, decode and encode. It credits more than 2,000 contributors.

Where it's strong

  • Coverage. The README claims support for 200+ model architectures from Hugging Face, including MoE, hybrid state-space, multimodal, embedding and reward models.
  • Hardware breadth. NVIDIA, AMD and Intel GPUs and x86, ARM and PowerPC CPUs are supported directly, with plugins for Google TPUs, Intel Gaudi, Huawei Ascend, Apple Silicon and others.
  • APIs. An OpenAI-compatible server plus Anthropic Messages API and gRPC support, tool-call and reasoning parsers, multi-LoRA, and structured outputs through XGrammar or Guidance.
  • KV cache plumbing. Automatic prefix caching skips recomputing shared prompt prefixes, and KV connectors such as NIXL, Mooncake, LMCache and a CPU offloading connector move cache data between instances or tiers.

What to watch

  • Disaggregated prefilling is still marked experimental in the vLLM docs, which also note that it does not improve throughput; it helps you tune TTFT and tail latency separately.
  • Releases land roughly every two weeks with breaking changes. The v0.33.0 release notes (release candidate as of Nov 14, 2026) say wheels and images are now CUDA 13.0 only, and the LM Format Enforcer structured-output backend is deprecated, due for removal in v0.35.0.

SGLang: built for prefix reuse

SGLang came out of LMSYS research on structured language model programs and is now a full serving framework. Its docs say it runs production deployments generating trillions of tokens a day across more than 400,000 GPUs; that is the project's own claim, not an audited figure.

Where it's strong

  • RadixAttention. SGLang keeps cached prefixes in a radix tree, so requests that share a system prompt, a document or earlier chat turns reuse that work automatically. This suits agent loops, where every step resends a long, mostly identical context.
  • Hierarchical caching. HiCache extends the prefix cache from GPU memory into host memory and external storage, which helps when the working set of conversations is bigger than VRAM.
  • Distributed serving. PD disaggregation splits prefill and decode onto different workers, with Mooncake and NIXL as transfer engines, plus data-parallel attention and a router for spreading load.
  • Structured outputs. JSON schema, regex or EBNF constraints, with XGrammar as the default backend and Outlines or llguidance as options (docs).
  • Hardware. The README lists NVIDIA, AMD Instinct, Google TPU (v6e, v7), Intel GPUs and Xeon CPUs, Apple Silicon, Huawei Ascend and Moore Threads, with more integrations in progress.

What to watch

  • The quickstart installs with uv pip install --prerelease=allow sglang, and the project ships a release about every two weeks. Pin versions and read the release notes before upgrading.
  • Model-specific launch flags matter. Use the SGLang Cookbook's recipe for your model and GPU rather than defaults.

TensorRT-LLM: NVIDIA's own engine

TensorRT-LLM is NVIDIA's library for LLM inference on its GPUs, with custom attention, GEMM and MoE kernels, prefill-decode disaggregation, wide expert parallelism and speculative decoding, according to its README.

The big change: no more engine builds

If your picture of TensorRT-LLM involves trtllm-build and per-model checkpoint conversion, it's out of date. NVIDIA's migration guide (last updated Nov 3, 2026) says the TensorRT engine backend has been removed: PyTorch is the only execution backend, and Hugging Face checkpoints load directly with trtllm-serve <model>. The 1.3 and 1.4 release candidates removed the remaining TensorRT-backend code. That makes TensorRT-LLM much closer to vLLM and SGLang in day-to-day use.

Where it's strong

  • NVIDIA-first optimizations. Quantization recipes include FP4, several FP8 variants, FP8 and NVFP4 KV caches and GPTQ/AWQ, and it runs pre-quantized checkpoints from NVIDIA Model Optimizer directly.
  • KV cache reuse by default. Blocks are stored in a radix search tree and reused across requests, with offloading to secondary memory and priority-based eviction (docs).
  • Serving features. trtllm-serve exposes OpenAI-style completions, chat and responses endpoints, /metrics, and tool and reasoning parsers. Guided decoding uses XGrammar or LLGuidance, switched on in the YAML config.

What to watch

  • NVIDIA only. The support matrix lists Blackwell, Grace Hopper, Hopper, Ada Lovelace and Ampere on Linux x86_64 or aarch64. There is no AMD, Intel or TPU path.
  • Release cadence. As of Nov 14, 2026, the latest stable GitHub release is v1.2.1 from April 2026, while current features ship in 1.3 and 1.4 release candidates, each with a list of breaking changes (releases). Plan to pin a release candidate or a container and test upgrades.

Feature comparison

Scroll the table horizontally on smaller screens.

Serving features compared
FeaturevLLMSGLangTensorRT-LLM
OpenAI-compatible serverYes, plus Anthropic Messages API and gRPCYes (chat, completions, vision, embeddings)Yes (models, completions, chat, responses)
Prefix / KV cache reuseAutomatic prefix cachingRadixAttention (radix tree)Block reuse in a radix search tree, on by default
KV offload beyond GPUCPU offloading and LMCache connectorsHiCache: host memory and storage tiersSecondary-memory offload with priority eviction
Prefill/decode disaggregationYes, marked experimental (NIXL, Mooncake, LMCache and others)Yes (Mooncake, NIXL)Yes, via trtllm-serve or Dynamo
Structured outputsXGrammar, GuidanceXGrammar (default), Outlines, llguidanceXGrammar, LLGuidance
Speculative decodingn-gram, suffix, EAGLE, DFlashYes, including adaptive speculative decodingYes
Quantization highlightsFP8, MXFP4, NVFP4, INT8/INT4, GPTQ, AWQ, GGUFFP8, AWQ, MXFP4 and more (hardware-dependent)FP4, FP8 variants, NVFP4 KV cache, GPTQ, AWQ
Multi-LoRA servingYesYesYes
Sources: vLLM README and docs, SGLang docs, TensorRT-LLM docs, accessed Nov 14, 2026. Feature depth varies by model and hardware; check each project's docs for your model.

Scroll the table horizontally on smaller screens.

Hardware support
HardwarevLLMSGLangTensorRT-LLM
NVIDIA GPUsYesYesYes (Ampere to Blackwell)
AMD GPUsYes (ROCm)Yes (Instinct MI300X to MI355X)No
IntelGPUs and CPUs; Gaudi via pluginArc GPUs, Xeon CPUsNo
Google TPUVia vllm-tpu pluginv6e, v7 (TorchTPU or JAX path)No
Huawei AscendVia pluginYesNo
Apple SiliconVia pluginYes (Metal / MLX)No
Sources: vLLM README and quickstart, SGLang README, TensorRT-LLM support matrix, accessed Nov 14, 2026.

Getting started: one model, three servers

The commands below are illustrative and were not run for this article (it needs a GPU host); they come from each project's quickstart as of Nov 14, 2026. Each starts an OpenAI-compatible server for the same Hugging Face model.

# vLLM: serves on http://localhost:8000 by default
uv pip install vllm
vllm serve Qwen/Qwen3-8B

# SGLang: Docker image, port 30000 as in the quickstart
docker run --rm --gpus all --ipc=host -p 30000:30000 \
lmsysorg/sglang:latest \
sglang serve Qwen/Qwen3-8B --host 0.0.0.0 --port 30000

# TensorRT-LLM: inside NVIDIA's release container; no engine build
trtllm-serve Qwen/Qwen3-8B

Because all three speak the same API, switching engines is a URL change in your client:

from openai import OpenAI
# Point base_url at whichever engine you are testing
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")
r = client.chat.completions.create(
model="Qwen/Qwen3-8B",
messages=[{"role": "user", "content": "Summarize PagedAttention in one line."}],
)
print(r.choices[0].message.content)

That portability is the main reason not to over-think the first choice. Keep engine-specific request fields, such as structured-output parameters, behind a thin wrapper so you can rerun the same workload on another engine later. If you're still deciding where the GPUs come from, see our comparisons of serverless GPU inference and spot vs reserved GPUs.

What about performance?

There is no neutral, current, apples-to-apples benchmark of the three engines that holds across models and GPUs. Each project publishes its own results, usually on a model and configuration it has tuned, and the numbers change with every release. Vendor charts show what is possible on a tuned setup, not what you will see on yours.

Results also flip with workload shape. Prefix-heavy agent traffic favors whichever engine gets the most cache hits; short independent prompts at high concurrency stress batching and decode kernels instead; and a new MoE model may be well-optimized in one engine weeks before the others. So treat any single "X is 2x faster" claim as a starting hypothesis.

The reliable approach is to run the same client, dataset and load sweep against each engine on your hardware and compare goodput at your latency targets. Our guide to benchmarking LLM inference covers the method and tools, including vLLM's and SGLang's built-in benchmark clients, which can both target the other engines.

How to choose

Decision flow for choosing between vLLM, SGLang and TensorRT-LLM by hardware, traffic and tuning needs
A starting-point decision flow. Benchmark before you commit. Graphic: Metarix, Nov 14, 2026.

Scroll the table horizontally on smaller screens.

Pick by situation
If you...Start withWhy
Need AMD, TPU, Intel or other non-NVIDIA hardwarevLLM or SGLangTensorRT-LLM supports NVIDIA GPUs only
Run agents or multi-turn chat with long shared contextSGLangPrefix reuse is its core design; HiCache adds memory tiers
Want the most models and community answersvLLM200+ architectures and 2,000+ contributors per its README
Run an all-NVIDIA fleet and need peak tokens per GPUTensorRT-LLM (benchmark against the others)NVIDIA's kernels and FP4/FP8 recipes
Plan multi-node, disaggregated servingAny, behind NVIDIA Dynamo or llm-dDynamo works with all three engines
Metarix editorial guidance based on project documentation, Nov 14, 2026.

You don't have to pick only one

Orchestration layers increasingly treat the engine as a plug-in. NVIDIA Dynamo says it works with vLLM, SGLang and TensorRT-LLM and adds disaggregated serving, KV-aware routing and autoscaling on top; SGLang also lists llm-d and Ray Serve as deployment options. Some teams run one engine for most models and another where it is clearly faster for a specific model.

Common mistakes

  • Following old TensorRT-LLM tutorials. Engine builds and convert_checkpoint.py scripts are gone from current releases. Use trtllm-serve with a Hugging Face checkpoint.
  • Comparing engines at default settings. Defaults differ in memory fraction, batch limits, prefix caching and warmup. Tune each one, or at least use each project's recommended recipe for your model.
  • Benchmarking with random prompts when production shares prefixes. You'll miss the cache effects that may decide the winner. Replay realistic traffic.
  • Upgrading blindly. All three ship fast and break things. Pin a version or container digest, and run your eval and load test before rolling forward.
  • Exposing the server directly. Treat these servers as internal services and put an API gateway with authentication and rate limits in front of any public endpoint.

Verdict

For most teams, vLLM is the right first engine: it runs on the most hardware, supports the most models and has the largest community, so you'll spend less time blocked. Choose SGLang when your workload is agentic or multi-turn and prefix reuse dominates, or when its hierarchical cache fits your memory problem; it's a peer to vLLM, not a niche tool. Choose TensorRT-LLM only when you are committed to NVIDIA hardware and have measured a gain worth the faster churn of its release candidates.

Because all three share the same API, the decision is reversible. Pick one, wrap it behind a stable client, and rerun a proper benchmark whenever a major release or a new model lands.

Sources

Tools covered in this guide

Frequently asked questions

Is SGLang faster than vLLM?
It depends on the model, hardware and traffic. SGLang is designed around reusing cached prefixes, so it often suits agentic and multi-turn workloads, while vLLM is often as fast on other workloads. Both projects publish their own benchmarks, which aren't directly comparable. Run the same load test on your own model and GPUs before deciding.
Does TensorRT-LLM still require building a TensorRT engine?
No. NVIDIA's migration guide, last updated Nov 3, 2026, says the TensorRT engine backend has been removed and PyTorch is the only execution backend. Hugging Face checkpoints load directly with trtllm-serve or the Python LLM API, with no trtllm-build or checkpoint conversion step.
Can vLLM, SGLang and TensorRT-LLM run on AMD GPUs?
vLLM and SGLang support AMD GPUs; SGLang lists Instinct MI300X through MI355X, and vLLM supports ROCm. TensorRT-LLM supports NVIDIA GPUs only.
Are vLLM, SGLang and TensorRT-LLM free?
Yes. All three are open source under the Apache 2.0 license. You pay for the GPUs or cloud instances they run on, plus any managed platform you use to host them.
Can I switch inference engines without changing my application?
Mostly. All three expose OpenAI-compatible chat and completions endpoints, so switching is usually a base URL change. Engine-specific request options, such as structured-output parameters and server flags, differ and need testing.

More in Hosting & inference

Share