Skip to content

Cloud & Architecture · Free

vLLM

vLLM is an open-source library for large language model inference and serving. It was originally developed in the Sky Computing Lab at UC Berkeley and is now maintained by a large community of academic and industry contributors under the Apache 2.0 license. Many teams use it as the engine behind self-hosted model endpoints, and several managed inference platforms build on it.

Its core idea is efficient memory management for the attention key-value cache, introduced as PagedAttention, combined with continuous batching of incoming requests, chunked prefill and prefix caching. The project README lists optimized attention and matrix-multiply kernels, CUDA and HIP graph execution, torch.compile-based graph transformations, speculative decoding methods and disaggregated prefill, decode and encode. It supports a long list of quantization formats, including FP8, INT8, INT4, GPTQ, AWQ and GGUF.

vLLM exposes an OpenAI-compatible API server, plus Anthropic Messages API and gRPC support, so existing client code can usually point at it with a URL change. It supports streaming, structured outputs, tool calling and reasoning parsers, multi-LoRA serving, and tensor, pipeline, data, expert and context parallelism for models that span several GPUs. Hardware support covers NVIDIA, AMD and Intel GPUs and x86, ARM and PowerPC CPUs, with plugins for accelerators such as Google TPUs and Intel Gaudi. The README says it supports more than 200 model architectures from Hugging Face, including dense and mixture-of-experts LLMs, multimodal models and embedding models.

For performance work, vLLM ships a benchmarking CLI. The vllm bench serve command load-tests a running server with configurable datasets, request rates, burstiness, concurrency caps and warmup requests, reports percentiles for time to first token, time per output token and inter-token latency, and can compute goodput against latency targets.

Who it is for: developers and platform teams who want to serve open-weight models on their own GPUs or cloud instances with high throughput.

Pricing: free and open source; you pay only for the hardware it runs on. Alternatives include SGLang, NVIDIA TensorRT-LLM and Hugging Face Text Generation Inference. Official repository: https://github.com/vllm-project/vllm

Official site