Skip to content

Cloud & Architecture · Free

SGLang

SGLang is an open-source serving framework for large language models, vision-language models and diffusion models. It is hosted by LMSYS, a non-profit open-source organization, and released under the Apache 2.0 license. Its documentation describes it as an inference framework for production serving, built for low latency and high throughput on anything from a single GPU to large distributed clusters, and it highlights agentic workloads, large-scale serving and reinforcement-learning rollouts as core use cases.

SGLang's best-known idea is RadixAttention: the server keeps the attention key-value (KV) cache of earlier requests in a radix tree, so a new request that starts with the same tokens, such as a shared system prompt, a long document or the earlier turns of a chat, reuses that cached prefix instead of recomputing it. HiCache extends this into a hierarchical cache that can spill KV data from GPU memory to host memory and external storage. The documentation also covers prefill-decode (PD) disaggregation with Mooncake or NIXL as the transfer engine, speculative decoding, LoRA serving, quantization (including FP8, AWQ and MXFP4 on supported hardware), a quantized KV cache, data-parallel attention and a router for spreading traffic across replicas.

For application developers, SGLang exposes OpenAI-compatible APIs for chat completions, completions, vision and embeddings, plus tool-call and reasoning parsers. Structured outputs can be constrained by a JSON schema, a regular expression or an EBNF grammar, with XGrammar as the default grammar backend and Outlines or llguidance as options. A model is started with the sglang serve command or the lmsysorg/sglang Docker image, and the project's Cookbook gives ready-made launch commands per model and hardware.

Hardware support listed in the README covers NVIDIA GPUs (from RTX cards to Blackwell systems), AMD Instinct GPUs, Google TPUs, Intel GPUs and Xeon CPUs, Apple Silicon, Huawei Ascend NPUs and Moore Threads GPUs. SGLang also works as a backend for orchestration layers such as NVIDIA Dynamo, llm-d and Ray Serve, and ships SGLang Diffusion for image and video generation.

Who it is for: platform teams and developers serving open-weight models on their own GPUs, especially for multi-turn, agentic or prefix-heavy traffic where cache reuse matters.

Pricing type: free and open source; you pay only for the hardware. Alternatives include vLLM, NVIDIA TensorRT-LLM and Hugging Face Text Generation Inference. Official repository: https://github.com/sgl-project/sglang (documentation at https://docs.sglang.io).

Official site