What is the best benchmark for AI agents?

There is no single best one, because agents do different jobs. τ-bench suits rule-following customer-service agents, MCPMark suits agents that use MCP servers, BrowseComp and GAIA suit web research, OSWorld suits computer use, and SWE-bench and Terminal-Bench suit coding and shell work.

What does pass^k mean in agent benchmarks?

Pass^k measures whether an agent succeeds on all k repeated attempts at the same task, introduced in the τ-bench paper. It is stricter than pass@1 and shows consistency, which matters when an agent acts on real customer or business data.

What is the difference between BrowseComp and GAIA?

BrowseComp has 1,266 hard web-search questions with short answers and measures persistence in finding obscure facts. GAIA has 466 assistant questions that can also require files, multiple modalities and tools, scored by quasi exact match on the final answer.

Can I compare scores across benchmark versions?

Usually not. The τ-bench and MCPMark documentation each say results from earlier versions are not comparable, and OSWorld 2.0 is a separate benchmark from OSWorld 1.0. Always check the version, harness and date next to a score.

Do benchmark scores predict how an agent will perform in my product?

Only loosely. Public benchmarks do not use your tools, policies or data, so use them to shortlist models and then build your own evals on tasks from your product.

Benchmarks · Comparison

AI Agent Benchmarks Compared: Which Ones Matter in 2026

Agent benchmarks test very different jobs, from rule-following and MCP tool use to web research and computer use. Here is what each one measures and how to read its scores.

Metarix Editorial · Nov 6, 2026 · 10 min read

On this page

There is no single best benchmark for AI agents, because "agent" covers very different jobs. τ-bench tests whether an agent follows rules while talking to a user, MCPMark tests real MCP tool use, BrowseComp and GAIA test web research, OSWorld tests computer use, and SWE-bench and Terminal-Bench test coding and shell work. Pick the one that matches the work your agent will do, then check its version, harness and reliability setting before you trust a number.

This guide compares the main agent benchmarks side by side: what each measures, how it is graded, how big it is, and where it misleads. It does not rank models. For coding benchmarks in depth, see our SWE-bench vs Terminal-Bench explainer.

The short answer: match the benchmark to the job

Scroll the table horizontally on smaller screens.

Agent benchmarks at a glance
Benchmark	What it tests	How it is graded	Size	Main caveat
τ-bench	Policy-following customer-service agents that talk to a simulated user and call tools	Final database state compared with a goal state; pass^k for repeat reliability	Airline, retail, telecom, banking and intake domains; 50 frozen tasks per domain in text-v1	The user is another model, and 2.0 scores are not comparable with 1.x
MCPMark	Real use of Notion, GitHub, filesystem, Postgres and Playwright MCP servers	Programmatic verifier per task; pass@1 and pass^4	127 tasks in the original paper	Needs credentials per service; only Verified-set scores are current
BrowseComp	Persistence in finding hard-to-find facts on the open web	Short answer checked against a reference	1,266 questions	Narrow skill; short answers only
GAIA	General assistant questions needing browsing, files and tools	Quasi exact match on a short final answer	466 questions; answers to 300 withheld	Grades only the final answer; English only
OSWorld	Operating a real desktop through screenshots, mouse and keyboard	Custom script checks final machine state	369 tasks (OSWorld 1.0); 108 in OSWorld 2.0	Heavy infrastructure; versions are not interchangeable
SWE-bench / Terminal-Bench	Fixing GitHub issues / finishing shell tasks	Hidden unit tests / task tests on container state	Varies by split and version	Contamination, broken tasks and harness effects

Project pages, repositories and papers listed under Sources, accessed November 6, 2026. Task counts change between versions, so check the version before comparing.

Tool use under rules: τ-bench and the Berkeley leaderboard

τ-bench is the closest thing to a customer-service agent test. Each task gives the agent a written policy and domain tools, plus a simulated user with a goal, such as changing a flight or returning an order. The benchmark compares the database at the end of the conversation with the expected state and checks that required information was communicated. According to the original paper, even strong function-calling agents of mid-2024 succeeded on fewer than half of tasks and were inconsistent across trials, which is why it introduced pass^k, a score that asks whether the agent succeeds on every one of k repeated attempts.

Pass^k is the metric to look for when you care about reliability. A model that solves a task in one run out of two has a respectable pass@1 on that task and a poor pass^2. If your agent will refund money or change records, the second number is closer to what your customers experience.

The project has changed a lot, and the README is explicit about it. Version 1.0 added full-duplex voice and a banking_knowledge retrieval domain, and version 1.0.1 fixed grading on that domain, so earlier banking results are not comparable. Version 2.0 adds the same airline, retail and telecom tasks in six languages in text and voice, plus an intake domain with 200 tasks for capturing names and addresses exactly. The README says 2.0 scores are not comparable with the 1.x leaderboard, and the site shows them on separate tabs.

The Berkeley Function Calling Leaderboard covers a lower layer: whether a model produces correct function calls. Its releases moved from single, parallel and multiple calls checked against an abstract syntax tree, to enterprise and community-contributed live data, to multi-turn and multi-step calls, and in version 4 to agentic web search, memory management and format sensitivity. The web search category needs a SerpAPI key to run. Use it to screen models for basic tool-call accuracy before you test full workflows.

Real MCP tools: MCPMark

MCPMark is the benchmark to read if your agent talks to MCP servers. Its tasks run against Notion, GitHub, the filesystem, Postgres and Playwright in isolated sandboxes, and a script verifies the final state. The paper describes 127 tasks that involve create, read, update and delete operations, and reports that models needed an average of 16.2 turns and 17.4 tool calls per task. In the paper, the best model at the time, GPT-5 at medium effort, reached 52.56% pass@1 and 33.86% pass^4.

The versioning rule matters here too. The README says MCPMark Verified became the default task set on June 12, 2026, with every environment version pinned and every verifier stabilised, and that earlier results are deprecated and not directly comparable. It also pins server versions, for example GitHub MCP Server v0.15.0 run through Docker. If a launch chart quotes an MCPMark number without saying which task set it used, treat it as unchecked.

If you are choosing servers rather than models, our guide to the best MCP servers for developers covers the official picks, and the MCP server security checklist covers what to review before you connect one.

Web research: BrowseComp and GAIA

These two get mixed up, but they test different things. BrowseComp, released by OpenAI on April 10, 2025, has 1,266 questions built to be hard to find but easy to verify, with a short single answer. OpenAI says it tests persistence and creative searching, and it states plainly that it is an incomplete benchmark, because short answers do not show how an agent does with open-ended requests.

OpenAI also reported how hard the questions are for people. In its verification campaign, human trainers solved 29.2% of problems within about two hours, and 70.8% were abandoned as unsolvable in that time. In the same post, a Deep Research agent scored 51.5%, but OpenAI notes that model was trained on data that specifically teaches BrowseComp-style tasks, so that number shows what targeted training can do.

GAIA, from researchers at Meta, Hugging Face and AutoGPT, has 466 questions that are simple for people but need browsing, files, multiple modalities and tool use. Answers to 300 are withheld for a leaderboard, and scoring is a quasi exact match on a short final answer. The paper reports 92% for human respondents against 15% for GPT-4 with plugins at the time of writing. Its authors list the main limits themselves: it does not grade the path to the answer, web evidence can change, and all questions are in English.

Use BrowseComp when your product is a search or research agent and you want a hard test of persistence. Use GAIA when you want a broader assistant test that includes files and calculations. Neither says anything about writing a good report from what the agent finds.

Computer use: OSWorld

OSWorld puts an agent in a real virtual machine with ordinary apps. The original benchmark has 369 tasks, each with an initial-state configuration and a custom script that checks the final state of the machine. The project says 8 Google Drive tasks can fail to initialise because of network and account issues, so results may be reported on 369 tasks or on 361, as long as the report says which. In July 2025 the team released OSWorld-Verified, which fixed community-reported issues and added AWS support so a full run can finish in about an hour.

OSWorld 2.0 arrived on June 26, 2026 and is a different test, not a patch. Its paper describes 108 long-horizon workflows across 31 self-hosted websites and desktop applications, with a median human completion time of about 1.6 hours and an average of 318 tool calls for Claude Opus 4.7 at maximum thinking, compared with about 30 in OSWorld 1.0. The authors report that the best configuration they tested, Claude Opus 4.8 with maximum thinking, completed 20.6% of tasks outright at a 54.8% partial score. Their analysis says agents mostly lose track of constraints, miss information that arrives mid-task and skip verification, rather than failing at basic clicking.

The repository lists osworld-v2.1, released September 16, 2026, as the recommended version. That is a good example of why you should always read the version string next to a score: a result on the 2026.06.24 release and one on v2.1 are not guaranteed to match.

Coding and terminal work: a short pointer

SWE-bench grades patches against a repository's tests, and Terminal-Bench grades whole shell tasks by checking the container's end state. Both have had their headline numbers questioned in 2026. OpenAI said on February 23, 2026 that it would stop reporting SWE-bench Verified because of broken tasks and signs of training-data exposure. Terminal-Bench 4.0's public leaderboard lists resolution rate with a 95% confidence interval, plus cost and tokens, which is the kind of reporting you want to see. Our SWE-bench vs Terminal-Bench explainer goes through both in detail.

How to read any agent benchmark score

Whatever the benchmark, run through the same checks before a number changes a decision. The pattern below is also what our guide to reading LLM leaderboards recommends for model charts.

Which version? τ-bench 2.0, MCPMark Verified and OSWorld 2.0 are new versions, and the τ-bench and MCPMark documentation says scores from earlier versions are not comparable.
Which harness? The agent scaffold, tools, step limits and timeouts can move a result as much as the model. Compare models only inside the same harness.
One run or many? Look for pass^k or repeated trials. Single-run scores hide inconsistency.
Is there a private or held-out split? GAIA withholds answers to 300 questions, and BrowseComp asks that its examples stay out of training data. A public split alone invites contamination.
What does it cost? Long agent runs can use hundreds of tool calls. A score without cost or token counts is half the picture.
Who ran it? Vendor-run numbers are useful but self-reported. Prefer maintainer-verified or independently reproduced results, and say which kind you quote.

Most important, no public benchmark knows your tools, your policies or your data. The benchmarks above tell you which models are worth testing. Your own evals, built with something like Promptfoo or Inspect, tell you which one to ship.

Which benchmark for which agent

Scroll the table horizontally on smaller screens.

Pick by what you are building
If you are building	Start with	Then add
Customer support or back-office agent with rules	τ-bench (pass^k)	Your own policy tests
Agent that uses MCP servers	MCPMark Verified	Function-calling checks from BFCL; your own server tests
Research or search agent	BrowseComp	GAIA for files and mixed tools
Agent that drives a desktop or browser UI	OSWorld 2.0	Your own end-to-end flows
Coding agent	SWE-bench private or held-out splits and Terminal-Bench	Repository-specific evals

Recommendation by Metarix Editorial based on what each benchmark's documentation says it measures. Not an official ranking.

What the published numbers say, with dates

These figures come from each benchmark's own paper or page. They were produced with different models, harnesses and dates, so they are not directly comparable and are shown to illustrate difficulty, not to rank.

τ-bench paper (June 2024): strong function-calling agents succeeded on under 50% of tasks, and pass^8 was under 25% in retail.
GAIA paper: 92% for human respondents versus 15% for GPT-4 with plugins.
BrowseComp (OpenAI, April 10, 2025): 0.6% for GPT-4o, 9.9% for o1 and 51.5% for a Deep Research model trained on BrowseComp-style tasks.
MCPMark paper (September 2025): 52.56% pass@1 and 33.86% pass^4 for the best model tested.
OSWorld 2.0 paper (June 2026): 20.6% of tasks completed outright, with a 54.8% partial score, for the best configuration the authors tested.

Verdict

Choose by job, then verify the version. For rule-following agents, use τ-bench and read the pass^k column. For MCP tool use, read MCPMark Verified. For web research, BrowseComp is the hard test and GAIA the broader one. For computer use, OSWorld 2.0 is the current stress test, and for coding, trust held-out splits over public ones.

Do not pick a model from a single headline number. Use benchmarks to build a shortlist of two or three models, then run your own evals on your own tasks before you commit.

Sources

Tools covered in this article

Frequently asked questions

What is the best benchmark for AI agents?: There is no single best one, because agents do different jobs. τ-bench suits rule-following customer-service agents, MCPMark suits agents that use MCP servers, BrowseComp and GAIA suit web research, OSWorld suits computer use, and SWE-bench and Terminal-Bench suit coding and shell work.
What does pass^k mean in agent benchmarks?: Pass^k measures whether an agent succeeds on all k repeated attempts at the same task, introduced in the τ-bench paper. It is stricter than pass@1 and shows consistency, which matters when an agent acts on real customer or business data.
What is the difference between BrowseComp and GAIA?: BrowseComp has 1,266 hard web-search questions with short answers and measures persistence in finding obscure facts. GAIA has 466 assistant questions that can also require files, multiple modalities and tools, scored by quasi exact match on the final answer.
Can I compare scores across benchmark versions?: Usually not. The τ-bench and MCPMark documentation each say results from earlier versions are not comparable, and OSWorld 2.0 is a separate benchmark from OSWorld 1.0. Always check the version, harness and date next to a score.
Do benchmark scores predict how an agent will perform in my product?: Only loosely. Public benchmarks do not use your tools, policies or data, so use them to shortlist models and then build your own evals on tasks from your product.