Benchmarks & Deep Dives · Free
AIPerf
AIPerf is NVIDIA's open-source command-line tool for benchmarking generative AI models served by an inference server. It is developed in the ai-dynamo organization on GitHub under the Apache 2.0 license and distributed on PyPI as aiperf. NVIDIA's older GenAI-Perf tool now carries a notice that it is being phased out and points users to AIPerf for new benchmarking work.
AIPerf acts as a load generator and measurement client. A typical run uses the aiperf profile command with a model name, endpoint type, tokenizer, server URL and either a concurrency level or a request rate. It reports metrics such as time to first token, time to second token, inter-token latency, request latency, output token throughput per user, output token throughput and request throughput, with averages, minimums, maximums and percentiles. Results are exported to CSV and JSON, and the tool offers a real-time dashboard, a simple progress view or a headless mode.
Its documentation lists several load-control modes: fixed concurrency, request rate, request rate with a concurrency cap, gradual ramping, arrival patterns including Poisson and gamma, warmup phases and request cancellation for resilience tests. A large part of its feature set targets realistic traffic: deterministic trace replay, replay of published production traces, multi-turn and agentic coding traces for KV-cache benchmarking, prefix synthesis and fixed schedules. It supports OpenAI-style chat, completions, embeddings, audio and image APIs, plus NIM embeddings and rankings, and has tutorials for Hugging Face TGI, AWS SageMaker endpoints and IAM-protected AWS endpoints using SigV4 signing.
Optional extras send results to MLflow, OpenTelemetry or Weights and Biases. The architecture is multiprocess, with internal services communicating over ZeroMQ, so it can generate high request volumes from one machine.
Who it is for: teams running NVIDIA Dynamo, Triton, TensorRT-LLM, vLLM, SGLang or other OpenAI-compatible servers who need detailed, reproducible performance profiles, especially for agentic or multi-turn workloads.
Pricing: free and open source. Alternatives include GuideLLM, vLLM's vllm bench serve, SGLang's bench_serving and Hugging Face inference-benchmarker. Official repository: https://github.com/ai-dynamo/aiperf
