Benchmarks & Deep Dives · Free
GuideLLM
GuideLLM is an open-source benchmarking and evaluation tool for large language model inference, maintained under the vLLM project on GitHub and released under the Apache 2.0 license. Its README describes it as an SLO-aware platform for measuring how a model and serving setup behave under realistic workloads, so teams can size deployments, tune configurations and plan capacity before traffic arrives.
GuideLLM works as a client. You point it at any OpenAI-compatible endpoint or a vLLM-native server, choose a traffic profile and a dataset, and it sends requests while recording latency and token-level statistics. Profiles include synchronous, concurrent, throughput, constant rate, Poisson and sweep. A sweep runs a series of load levels to find the maximum rates a deployment can handle. Constraints can stop each run by duration, request count or error count, and an over-saturation detector can end a run once the server stops keeping up. Warmup and cooldown periods exclude start and shutdown effects from the results.
Reported metrics include full distributions for time to first token (TTFT), inter-token latency (ITL) and end-to-end request latency, along with request and token throughput. Results are written to JSON and CSV by default, and GuideLLM can also produce YAML, self-contained HTML reports and static plots. Built-in scenarios bundle common workload shapes, such as a concurrent workload with 1,000 prompt and 1,000 output tokens.
Data can come from synthetic text with fixed prompt and output token counts, Hugging Face datasets, or local JSON, CSV and text files. GuideLLM can also synthesize images and short videos to benchmark vision-language models, and it can target chat completions, text completions, embeddings and audio transcription endpoints.
Who it is for: platform and ML engineers who need repeatable load tests and shareable reports for self-hosted LLM serving. It installs with pip (Python 3.11 to 3.15, Linux or macOS) or runs as a multi-arch container image from GitHub's container registry.
Pricing: free and open source. Alternatives include NVIDIA AIPerf, vLLM's built-in vllm bench serve command, SGLang's bench_serving script and Hugging Face inference-benchmarker. Official repository: https://github.com/vllm-project/guidellm
