MCPMark

MCPMark is an open-source benchmark that tests how well AI models and agents use real Model Context Protocol (MCP) servers. Instead of mocked tools, its tasks run against Notion, GitHub, the filesystem, Postgres and Playwright, each in an isolated sandbox, and every task has a programmatic verifier that checks whether the end state is correct. The paper, posted on arXiv in September 2025, describes 127 tasks created by domain experts together with AI agents, with a curated starting state for each task and a wide range of create, read, update and delete operations rather than read-only lookups.

How scoring works. A minimal agent runs in a tool-calling loop, and a script verifies the outcome. Results are reported as pass@1, and as pass^4, which asks whether the agent solved the same task in all four runs. In the original paper the best model, GPT-5 at medium effort, reached 52.56% pass@1 and 33.86% pass^4, while models such as Claude Sonnet 4 and o3 were below 30% pass@1. On average, models needed 16.2 turns and 17.4 tool calls per task, which is why the benchmark describes itself as a stress test.

Versions. According to the repository README, MCPMark Verified became the default task set on June 12, 2026. In that set every environment version is pinned and every verifier is stabilised. The README says results from earlier task versions are deprecated and not directly comparable, and asks people to report new numbers as MCPMark Verified. It also notes pinned server versions, for example GitHub MCP Server v0.15.0 run through Docker and Notion MCP 1.9.1, and an easy set of 50 tasks, ten per server, for smaller open models. The README reports a top Verified score of 92.9% for GPT-5.5 at its highest reasoning setting; treat that as a project-reported figure and check the live leaderboard at mcpmark.ai before quoting it.

Who it is for. Developers who build agents on MCP servers, teams choosing a model for tool-heavy workflows, and MCP server authors who want to see whether their server helps agents finish tasks with fewer tokens.

Limits to keep in mind. You need credentials for each service you want to test, scores depend on the agent scaffold and the server versions, and five services are a narrow slice of the MCP ecosystem. The software is free to run and you pay for model usage. Alternatives include the Berkeley Function Calling Leaderboard for function-calling accuracy, τ-bench for policy-following conversations and Terminal-Bench for shell tasks.