Cloud & Architecture · Free
vLLM
vLLM is an open-source library for large language model inference and serving. It was originally developed in the Sky Computing Lab at UC Berkeley and is now maintained by a large community of academic and industry contributors under the Apache 2.0 license. Many teams use it as the engine behind self-hosted model endpoints, and several managed inference platforms build on it.
Its core idea is efficient memory management for the attention key-value cache, introduced as PagedAttention, combined with continuous batching of incoming requests, chunked prefill and prefix caching. The project README lists optimized attention and matrix-multiply kernels, CUDA and HIP graph execution, torch.compile-based graph transformations, speculative decoding methods and disaggregated prefill, decode and encode. It supports a long list of quantization formats, including FP8, INT8, INT4, GPTQ, AWQ and GGUF.
vLLM exposes an OpenAI-compatible API server, plus Anthropic Messages API and gRPC support, so existing client code can usually point at it with a URL change. It supports streaming, structured outputs, tool calling and reasoning parsers, multi-LoRA serving, and tensor, pipeline, data, expert and context parallelism for models that span several GPUs. Hardware support covers NVIDIA, AMD and Intel GPUs and x86, ARM and PowerPC CPUs, with plugins for accelerators such as Google TPUs and Intel Gaudi. The README says it supports more than 200 model architectures from Hugging Face, including dense and mixture-of-experts LLMs, multimodal models and embedding models.
For performance work, vLLM ships a benchmarking CLI. The vllm bench serve command load-tests a running server with configurable datasets, request rates, burstiness, concurrency caps and warmup requests, reports percentiles for time to first token, time per output token and inter-token latency, and can compute goodput against latency targets.
Who it is for: developers and platform teams who want to serve open-weight models on their own GPUs or cloud instances with high throughput.
Pricing: free and open source; you pay only for the hardware it runs on. Alternatives include SGLang, NVIDIA TensorRT-LLM and Hugging Face Text Generation Inference. Official repository: https://github.com/vllm-project/vllm

Benchmarks · Deep dive · Metarix Editorial · Nov 15, 2026 · 11 min
Long-Context Benchmarks Explained: RULER, NIAH, LongBench
Needle-in-a-haystack scores say little about how well a model uses a long context. Here is what RULER, NoLiMa, MRCR, LongBench v2 and HELMET each measure, and how to test your own documents.
Read the deep dive →
Cloud · Comparison · Metarix Editorial · Nov 14, 2026 · 11 min
vLLM vs SGLang vs TensorRT-LLM: Which to Use
The three leading open-source LLM inference engines compared on hardware, prefix caching, disaggregated serving and release churn, with setup commands and a clear pick for each situation.
View comparison →
Benchmarks · Guide · Metarix Editorial · Nov 12, 2026 · 11 min
How to Benchmark LLM Inference: Tools and Method
A repeatable method for load-testing LLM serving, the metrics to report (TTFT, ITL, goodput), and vllm bench serve, SGLang bench_serving, GuideLLM and AIPerf compared.
Read article →