Benchmarks · Guide
Cheap Model or Expensive Model? Measure Cost per Successful Task
Compare total cost divided by verified successes. Include failed attempts, retries, routing and tools before deciding that a cheaper token rate saves money.
Metarix Editorial · Nov 5, 2026 · 4 min read
Documentation-based research
On this page
The wrong denominator makes cheap models look expensive—or free
Price per million tokens is easy to compare. It is less useful when one model needs three attempts, another finishes once, and a third produces an answer that looks plausible but fails validation. For an application, success is the deliverable, not the number of responses generated.
Google’s November 3 developer article explores routing between Gemini models and thinking levels using a lightweight classifier. It reports its own task set and experiment. That is a useful design example, not evidence that the same routing rule will save money on your workload.
Scroll the table horizontally on smaller screens.
| Measure | Option A | Option B |
|---|---|---|
| Initial tasks | 100 | 100 |
| Cost per attempt | $0.01 | $0.02 |
| Additional retry attempts | 30 | 0 |
| Total spend | $1.30 | $2.00 |
| Verified successes | 60 | 95 |
| Cost per verified success | About $0.0217 | About $0.0211 |
Define success before choosing a router
Choose a task with an observable acceptance condition: valid extraction plus correct fields, a code change passing relevant tests, or an answer supported by the provided sources. Make the rubric independent of the candidate model’s self-confidence.
Separate syntax, factual correctness, completeness and permissions. A perfect JSON object with the wrong invoice total is not a successful extraction. An agent that passes tests by editing the test expectations has not necessarily fixed the defect. Decide how reviewers will identify such failures before running the comparison.
A worked scenario: cheaper calls, higher unit cost
Consider 100 tasks. Option A makes 100 initial calls at $0.01, then retries 30 tasks once at the same rate. Its total is $1.30 and it produces 60 verified successes: about $0.0217 per successful task. Option B makes 100 calls at $0.02 and produces 95 verified successes: about $0.0211 per success.
The more expensive call is slightly cheaper per success in this constructed scenario, and it leaves far fewer failures. These are illustrative numbers, not measured model results or current API prices. Human review, tool use and infrastructure would need adding to both totals where relevant.
What a fair cost ledger includes
- All attempts, including timeouts and invalid responses—not only the winning answer.
- Input, generated output, relevant reasoning billing and cache behavior.
- Search, execution, storage and other paid tools.
- Router or classifier cost and fallback cost.
- Latency, abandoned tasks and human review required to reach an acceptable result.
Route narrow decisions, then verify them
A router can use task type, required tools, data sensitivity and observed difficulty to pick a path. Avoid assuming that a short prompt is easy or a long one is hard. A tiny arithmetic problem may require careful reasoning while a long formatting request may be routine.
Start with explicit categories and an escalation policy. Send clearly supported routine cases to the cheaper route; escalate unsupported cases and repeated validation failures. Keep a maximum retry and spend budget. Preserve an audit trail showing why a task took a particular path without recording unnecessary private content.
Use a held-out task set
Build a small representative sample with known answers or reviewable outcomes. Use one portion to design routing rules and another to check them. If you tune repeatedly on the same examples, your apparent gains may not transfer. Record model identifiers, prompts, settings and the date because hosted models and billing can change.
Report success counts and sample size alongside averages. For high-impact actions, require the appropriate human approval even when a router predicts success. A probability estimate is not permission to send money, publish a message or modify production.
Decide whether routing is worth the complexity
Routing adds code, monitoring and failure modes. A single sufficiently capable model with a shorter workflow can be the right choice for low volume. At higher volume, a measured routing policy may justify its maintenance. First establish a baseline, then compare changes against the same rubric. We verified the scenario arithmetic locally; no paid model comparison was run for this article.
Sources and related reading
Official documentation checked 5 November 2026. Documentation-based research; illustrative scenarios are labeled. Product availability and pricing can change.
Frequently asked questions
- What is cost per successful task?
- Total attributable spend across the evaluated workload divided by tasks that meet a predefined acceptance condition.
- Are the example success rates model benchmarks?
- No. They are hypothetical inputs showing how the calculation works. Real comparisons require a recorded task set and actual outcomes.