LLM Latency and Throughput Explained

LLM speed claims mix four different clocks: time to first token, time to first answer token, output tokens per second, and end-to-end latency. If you pick the wrong one—or compare charts that measured different workloads—you can shortlist a model that looks fast on a leaderboard and feels sluggish inside a coding agent.

This guide defines those metrics the way independent benches and provider docs actually define them, explains why public speed numbers are not interchangeable, and shows which clock matters for agent loops versus batch jobs. It does not publish a new tokens-per-second ranking, and it does not invent provider scores.

The four numbers that matter

Artificial Analysis publishes a clear set of inference definitions on its methodology page (accessed November 7, 2026). Use those as a shared vocabulary even when you measure your own stack with your own prompts.

Time to First Token (TTFT)

TTFT is the time in seconds between sending a request and receiving the first token of the response. On multi-turn workloads, Artificial Analysis averages it over a session's turns. For reasoning models that return reasoning tokens, the first token is typically the first reasoning token—not the first token of the answer the user will read.

Time to First Answer Token

Time to first answer token is the wait until the first answer token. For reasoning models this includes time spent reasoning, so unlike raw TTFT it measures the wait for an answer the same way for reasoning and non-reasoning models. If your product hides chain-of-thought from the UI, this is usually closer to what the user experiences than TTFT alone.

Output speed (tokens per second)

Artificial Analysis defines output speed as the number of output tokens—including reasoning tokens—received per second over a request's full duration, from sending the request to receiving the final token. That means TTFT sits in the denominator. For multi-turn sessions, AA divides the session's output tokens by the time the endpoint spent answering its requests, and excludes the time tools take between turns.

OpenAI's latency-optimization guide discusses inference speed as TPM (tokens per minute) or TPS (tokens per second)—the rate at which the model processes tokens—and treats that rate as only one of seven levers for product latency. Provider marketing often highlights peak TPS under light load. AA's output speed is deliberately end-to-end customer experience on serverless endpoints, not a hardware peak on empty GPUs.

End-to-end time

End-to-end time is seconds from send to final token on each request. For a multi-turn session, Artificial Analysis totals the session's request times and again excludes tool runtime between turns. Your product dashboard should also log tool time separately if you care about full agent-loop wall clock.

One more vocabulary note from the same methodology: native tokens are tokens as counted by the model's own tokenizer, usually as reported by the provider API. Prices and AA's performance metrics both use native tokens. Comparing "words per second" or a third-party tokenizer's counts against native-token charts will quietly mis-rank endpoints.

Why leaderboard speed numbers are not interchangeable

Two providers can both say "fast tokens per second" and still describe different experiments. Before you trust a comparison, check the harness the same way you would for quality scores—our guide on how to read LLM leaderboards applies here too.

• Prompt and output length: Prefill cost grows with input; decode cost grows with output. A short greeting probe inflates tok/s relative to a multi-thousand-token patch or a long tool-using transcript.
• Region and network path: The same model name in a distant region adds round-trip time to TTFT without changing GPU decode speed.
• Concurrency and saturation: OpenAI notes that running engines at lower saturation can raise TPM; crowded shared endpoints often do the opposite. Peak single-stream numbers rarely match your production concurrency.
• Streaming vs non-streaming clients: Streaming changes when the client sees tokens. It does not by itself prove lower total compute time for the same generation.
• Whether TTFT sits inside tok/s: AA includes TTFT in output speed. Some vendor charts start the stopwatch at the first token. Mixing those two definitions will mis-rank the same stream.
• Reasoning tokens: Models that emit long chain-of-thought inflate output token counts and delay the first answer token. Ignoring that favors "thinking" models on raw tok/s and punishes them on time-to-answer.
• Tokenizer differences: Always prefer native tokens from the API usage fields over approximate word counts.
• Who ran the test: Aggregators like Artificial Analysis and routers like OpenRouter are useful for cross-provider views, but each has its own workload mix and sampling window. Read the methodology date and the endpoint under test.

Treat every public speed table as a shortlist input, not a purchase order. If the footnote does not state prompt size, output cap, region, concurrency, streaming mode, and whether TTFT is in the rate, you do not yet have a comparable number.

What coding agents actually feel

Agents do not feel abstract TPS. They feel wall-clock time to the next usable action: a streamed plan line in the IDE, a finished tool-call JSON blob, or a completed patch ready to apply. That maps unevenly onto the four metrics.

Short steps vs long generations

Many agent steps are short: classify intent, choose a tool, rewrite one failing assertion, emit a structured plan. Output is small, so TTFT and time-to-first-answer dominate perceived snappiness. A model that is slightly slower at sustained decode but much faster to first token often feels better in a tight loop.

Long refactors are the opposite. Once the model is emitting hundreds or thousands of tokens, sustained output speed and end-to-end time dominate. Saving tens of milliseconds of TTFT barely matters if the patch still takes tens of seconds to finish—and if the patch fails tests, you pay that latency again on retries.

Tool loops and hidden wait

In a tool loop, end-to-end time per model call stacks with tool runtime (tests, builds, browser actions, MCP servers). Artificial Analysis excludes tool time from multi-turn end-to-end on purpose, because it is measuring the endpoint. Your product latency includes tools. Log model time and tool time as separate fields or you will "optimize" the wrong half of the loop.

Speed without task success is a false win. A fast model that fails the task burns retries; a slower model that finishes once can win on cost per successful task. For which public agent benches measure which jobs, see AI agent benchmarks compared.

Streaming, batching, and concurrency

Streaming cuts perceived wait

Anthropic's Messages API lets you set stream: true and receive server-sent events so text, tool use, and thinking deltas arrive incrementally (docs accessed November 7, 2026). OpenAI's latency guide lists streaming as the single most effective way to make users wait less: perceived blank waiting drops to about a second instead of the full generation.

Streaming improves the user–app system latency—the reader finishes sooner—even when total tokens and GPU time are unchanged. Chunking helps when you must post-process output (moderation, translation): stream to your backend, then forward processed chunks to the client. Do not cite a streaming demo as proof that a provider's non-streaming end-to-end time is lower.

Batch and offline jobs

Batch scoring, overnight evals, and offline report generation care about tokens completed per dollar and per wall-clock hour, not TTFT. Optimize concurrency, use batch APIs where they exist, and cut unnecessary output. Interactive UX tricks are the wrong spend for a job nobody is watching.

Concurrency, saturation, and gateways

OpenAI notes that running engines at lower saturation can give a modest TPM boost. The production flip side is familiar: as you raise parallel requests on a shared endpoint, per-request TTFT and tok/s often degrade. Measure at the concurrency you will actually run at peak, after a warm-up window—not with a lonely curl from your laptop.

If you front multiple providers, gateways and routers add their own hop and retry policy. Our AI gateways compared piece covers that layer; account for gateway overhead when you log TTFT so you do not blame the model for a proxy timeout.

How to measure your own stack

Vendor charts are useful for a shortlist. Your harness decides the ship call. A minimal, honest checklist:

1. Fix the prompt and max tokens. Same system prompt, same user payload, same max_tokens or stop rules for every candidate.
2. Record TTFT and total time. Timestamp request send, first byte or first token event, and final token. Prefer the provider's streaming events when available.
3. Compute output tok/s two ways if needed. (a) total output tokens ÷ full duration (AA-style); (b) tokens after first token ÷ decode-only duration. Label which one you publish internally.
4. Test at your concurrency. Warm up, then run the parallelism your product uses at peak—not a single serial loop.
5. Separate cold vs warm. First call after idle can include cold-start or cache-miss effects; steady-state needs a warm window.
6. For reasoning models, track time-to-first-answer. Do not treat the first reasoning token as "the answer started" if the UI hides reasoning.
7. Log region and endpoint ID. Same model display name on another region or third-party host is a different experiment.
8. Never mix dashboards without a harness note. A screenshot plus a footnote beats a naked number in a slide deck.

If you route through a multi-provider gateway, pin the upstream endpoint in the log line. Otherwise A/B results will silently mix hosts the next time the router rebalances.

How teams cut latency without lying to themselves

OpenAI's latency-optimization guide (accessed November 7, 2026) groups practical work into seven principles. Summarized in our words, with credit to that guide:

1. Process tokens faster. Prefer a smaller model that still hits quality bars, or inference features that skip work you already know (including predicted outputs for known-structure edits).
2. Generate fewer tokens. Ask for brevity, shorten structured field names, stop early—OpenAI's heuristic is that cutting about half of your output tokens may cut about half of your latency.
3. Use fewer input tokens. Usually a smaller lever (their rough range: cutting half the prompt may only save about 1–5% latency) unless context is huge; still prune RAG noise and keep a stable shared prefix for caching.
4. Make fewer requests. Combine sequential LLM steps into one response when you can avoid extra round trips.
5. Parallelize. Fire independent calls together; use speculative execution when one classification outcome is likely.
6. Make users wait less. Stream, chunk post-processing, show intermediate steps and loading states.
7. Don't default to an LLM. Hard-code constrained confirmations, pre-compute when inputs are finite, and use ordinary UI or classical code where they are faster and more reliable.

For code-edit style tasks where most of the file is unchanged, OpenAI documents Predicted outputs: you supply a prediction of the likely output so the model can focus on the diff. That is a documented latency feature for known-structure edits, not a general speedup for open-ended chat.

On the input side, prompt caching cuts repeated prefix cost and can help latency when the cache hits. When caching pays off for your traffic shape is covered in prompt caching cost break-even.

The honest version of "we made it faster" always states which clock moved, under which prompt length and concurrency, and whether quality or cost per successful task got worse. A slide that only shows peak tok/s is marketing, not an engineering report.

Verdict

Pick the metric by the job. Interactive coding agents should optimize TTFT or time-to-first-answer for short steps, and output speed plus end-to-end time for long patches. Batch jobs should optimize throughput at real concurrency and cost per completed task.

Always pair speed with quality and cost per successful task. Treat vendor and aggregator charts as shortlists. Measure your own harness—fixed prompts, a labeled tok/s definition, your region, your concurrency—before you change the default model in production.

Sources

• Artificial Analysis, Benchmarking methodology (definitions: TTFT, time to first answer token, output speed, end-to-end time, native tokens) — accessed November 7, 2026
• OpenAI, Latency optimization guide (seven principles, TPM/TPS, streaming, saturation, predicted outputs) — accessed November 7, 2026
• Anthropic, Streaming messages (stream: true, SSE) — accessed November 7, 2026
• OpenAI, Predicted outputs — accessed November 7, 2026