Cloud & Architecture · Free
TensorRT-LLM
TensorRT-LLM (written TensorRT LLM in NVIDIA's current documentation) is NVIDIA's open-source library for optimizing large language model and visual generation inference on NVIDIA GPUs. The README lists it under the Apache 2.0 license. It combines custom kernels for attention, matrix multiplication and mixture-of-experts layers with runtime techniques such as prefill-decode disaggregation, wide expert parallelism and speculative decoding.
The project has changed shape over the past year. Older versions compiled each model into a TensorRT engine with a separate trtllm-build step. Current releases are architected on PyTorch, and the documentation's migration guide (last updated November 3, 2026) states that the TensorRT engine backend has been removed: Hugging Face checkpoints load directly, with no checkpoint conversion or engine build. Teams following older tutorials should expect commands like trtllm-build to be gone.
For serving, the trtllm-serve command starts an OpenAI-compatible server with /v1/models, /v1/completions and /v1/chat/completions endpoints, plus a /metrics endpoint for runtime statistics. Options are passed on the command line or in a YAML configuration file, and the server supports tool-call and reasoning parsers. The Python LLM API covers offline and programmatic use from a single GPU to multi-node deployments, and integrates with NVIDIA Dynamo and Triton Inference Server.
Key features documented include KV cache reuse across requests (enabled by default, using a radix search tree with offloading to secondary memory and priority-based eviction), guided decoding with XGrammar or LLGuidance, disaggregated serving, and quantization recipes including FP4, several FP8 variants, FP8 and NVFP4 KV caches and GPTQ or AWQ weight-only formats. NVIDIA publishes pre-quantized checkpoints made with its Model Optimizer that run directly.
Hardware and platform: Linux on x86_64 or aarch64 with NVIDIA GPUs. The support matrix lists Blackwell, Grace Hopper, Hopper, Ada Lovelace and Ampere architectures; other architectures rely on community support. It does not target AMD, Intel or TPU hardware.
Who it is for: teams running on NVIDIA data-center GPUs that want NVIDIA's own optimizations, often alongside Dynamo for multi-node serving. NVIDIA ships frequent release candidates; check the release notes for breaking changes before upgrading.
Pricing type: free and open source. Alternatives include vLLM, SGLang and Hugging Face Text Generation Inference. Official repository: https://github.com/NVIDIA/TensorRT-LLM (documentation at https://nvidia.github.io/TensorRT-LLM/).
