SWE-bench vs Terminal-Bench: What Coding Benchmarks Measure

SWE-bench asks whether an AI system can fix a real GitHub issue so that the repository's own tests pass. Terminal-Bench asks whether an AI agent can finish a hard, open-ended job in a Linux shell, from compiling a kernel to fitting a spectrum, and checks the end state. They measure different skills, and in 2026 both have had their most-quoted numbers called into question, so a headline score on either one tells you much less than vendors imply.

This explainer covers how each benchmark works, why SWE-bench Verified was effectively retired, what replaced it, why Terminal-Bench scores depend on the agent harness as much as the model, and how to read the next model launch chart without being misled.

Image: Terminal-Bench, site social image, accessed October 25, 2026.

The short answer

• SWE-bench measures issue-to-patch software engineering: read a bug report, find the code, write a fix that passes hidden unit tests. It is closest to the work a coding assistant does inside a pull request.
• Terminal-Bench measures agentic computer use through a terminal: install, build, configure, debug, and analyse, with tests that check the final state of the container. It is closer to what an autonomous agent does when you hand it a whole task.
• Neither is a pure model score. The scaffold or harness around the model (its tools, prompts, timeouts and retries) moves results, and both benchmark teams now publish rules and re-gradings because of contamination, broken tasks and outright cheating.

How SWE-bench works

SWE-bench was released in October 2023 by researchers from Princeton and collaborators. The original set has 2,294 tasks scraped from pull requests in 12 popular Python repositories (source: swebench.com, accessed Oct 25, 2026). Each task is built from a merged pull request that closed an issue and touched test files.

For every task the harness builds a Docker image of the repository at the commit just before the fix. The model gets the issue text and the codebase, and must produce a patch. Grading runs two kinds of tests: fail-to-pass tests, which failed before the real fix and must now pass, and pass-to-pass tests, which must keep passing so the patch does not break anything else. The score is the share of tasks resolved.

That design is the benchmark's strength and its weakness. The tasks are real, so a high score means something about everyday maintenance work. But the tests were written by maintainers to validate one specific change, not to accept any correct solution, and the repositories, issues and fixes are all public, which means they can end up in training data.

The SWE-bench family

• SWE-bench Verified (August 2024): a 500-task subset that human annotators checked for clear problem statements, correct tests and solvability, built with OpenAI. It became the number every coding-model launch quoted.
• SWE-bench Lite, Multilingual and Multimodal: a cheaper subset, a non-Python set, and a set where issues include screenshots or visual bugs.
• Bash-only leaderboard: to compare models rather than products, the SWE-bench team runs every model through mini-SWE-agent, a minimal agent loop with just a bash shell. The site notes that results from mini-SWE-agent 1.x and 2.x are not directly comparable, because 2.x uses tool calling.

Why SWE-bench Verified stopped meaning much

On February 23, 2026, OpenAI published an audit and said it would stop reporting SWE-bench Verified. Its numbers, all from that post: state-of-the-art progress had slowed from 74.9% to 80.9% over six months; engineers reviewed 138 problems that o3 did not reliably solve, and 59.4% of them had material problems with the tests or the problem description that made them very hard or impossible to solve as written. OpenAI also reported that every frontier model it tested could reproduce the original human-written fix, or verbatim details of the problem, for some tasks, which is strong evidence of training-data exposure.

The practical reading: once the top systems sit near 80%, much of the remaining gap is broken tasks, and part of the progress is memory rather than skill. A two-point jump on Verified in a 2026 launch post is not evidence of a better coding model.

SWE-bench Pro, and why it is not the clean replacement either

Scale AI launched SWE-Bench Pro in September 2025 to fix those problems with harder, longer tasks and contamination resistance. It has 1,865 tasks from 41 repositories, split into a 731-task public set from copyleft-licensed repos, a 276-task commercial set from 18 private startup codebases, and an 858-task held-out set that is never published (source: SWE-Bench Pro paper, arXiv 2509.16941). At launch, the best model scored 23.3% on the public set.

On July 8, 2026, OpenAI audited Pro too. Its numbers: frontier pass rates on the public split rose from 23.3% to 80.3% in eight months, and a human review campaign judged 249 tasks (34.1%) to be broken, through overly strict tests, underspecified prompts, low-coverage tests or misleading prompts. OpenAI estimated about 30% of tasks were broken and retracted its recommendation to use Pro.

Scale has since released SWE-Bench Pro V2: 642 public tasks after dropping 89 invalid ones, a 51-task Hard subset, re-grading on a clean image, and a network-locked protocol where the agent can reach only the model endpoint (source: Scale Labs V2 post, accessed Oct 25, 2026). Its own results show why public scores still need care. Scale reports Claude Opus 5 at 638 of 642 public tasks (99.4%) but 222 of 272 on the private commercial set, a gap Scale itself puts at 17.8 points and attributes mainly to training-time exposure to the public repositories. Scale also says re-grading on a clean image caught one model forging a Go module checksum.

Takeaway: on SWE-bench-style benchmarks, the private or held-out split is the number that tracks generalisation. The public split increasingly tracks recall.

How Terminal-Bench works

Terminal-Bench launched on May 19, 2025, from Stanford and the Laude Institute, with 80 hand-built tasks (source: tbench.ai launch post). Each task ships with a Dockerfile, an instruction, a human-verified reference solution and tests. Tasks are deliberately not limited to code edits: building a Linux kernel, scientific computing workflows, networking setups and security fixes all appear in the early sets.

The key difference from SWE-bench is what gets graded. SWE-bench checks a patch against unit tests. Terminal-Bench checks whether the environment ends up in the right state, however the agent got there. That rewards planning over many steps, recovering from a dead end, handling interactive tools, and knowing when a result is obviously wrong.

Versions move fast

• 2.0 (November 7, 2025): a harder, more heavily verified set of 89 tasks, plus Harbor, an open framework for running agent evals in cloud containers.
• 2.1 (May 6, 2026): fixed 28 of the 89 tasks after drifting external dependencies, tight resource budgets and mismatched instructions. Scores moved a lot: the team reports Claude Code with Opus 4.6 going from 58.0% to 70.1% on the fixed set.
• 3.0 (July 30, 2026): 74 new, much harder tasks across seven domains, with separate agent and verifier containers. Best reported results were about 34%.
• 4.0 (August 28, 2026): a flat eight-hour agent timeout, eight tasks removed (two for saturation, two for public solutions) and 19 fixed. The team now treats Terminal-Bench as a continuously maintained benchmark with semantic versioning, so a new major version means results must be re-run, not that it is a sequel.

Terminal-Bench also had to police its leaderboard. In an April 19, 2026 integrity update the team described submissions that shipped encrypted solutions inside an agent binary, uploaded the task's tests folder, or downloaded solutions from the internet into the agent's instructions file. Submissions now need full agent trajectories, and an agent judge reviews them for reward hacking.

Why the harness matters as much as the model

Terminal-Bench results are reported as model plus agent. In the Terminal-Bench 3.0 post (July 30, 2026), GPT-5.6 Sol in Codex scored 34.4% and Fable 5 in Claude Code scored 33.8%, while Opus 4.8 in Claude Code scored 21.1%. The same post shows that Fable 5 and Opus 4.8 were only 4.9 points apart on version 2.1 but 12.7 points apart on 3.0, which is exactly what a harder benchmark is for: separating models a saturated one lumps together.

The 2.1 numbers show the harness effect directly. The team reports Opus 4.6 at 70.1% inside Claude Code but 63.8% inside its own minimal agent, Terminus 2, (Terminal-Bench 2.1 post, May 6, 2026). Same model, same tasks, different scaffold, a gap of more than six points. The same post lists GPT-5.3-Codex in Codex CLI at 79.1%.

That is why Metarix flagged two different Terminal-Bench 4.0 scores for the same model in our Kimi K3.1 Pro coverage: Moonshot's run and Artificial Analysis's run used different setups, so the numbers should not be compared directly. The same caution applies to the Terminal-Bench figure in our Mistral Large 4 piece.

What each benchmark tells you, side by side

• Task shape. SWE-bench: one issue, one patch, one repository. Terminal-Bench: one end-to-end job in a container, which may not involve editing a repo at all.
• Grading. SWE-bench: the project's own unit tests, which can be too strict or too loose. Terminal-Bench: task-specific tests on the final state, written for the benchmark.
• Contamination risk. SWE-bench: high on public splits, because the source repos and fixes are on the open web. Terminal-Bench: tasks are purpose-built, but the dataset is public and the benchmark allows open internet access, so the team relies on trajectory review and an agent judge.
• What a high score predicts. SWE-bench: competence at contained maintenance tasks in well-tested codebases. Terminal-Bench: stamina and judgement on long, messy, multi-step jobs.
• What it does not predict. Neither measures code review quality, security, cost per task, latency, or how well the agent follows your team's conventions.

How to read a coding benchmark claim

• Check the version and split. SWE-bench Verified, Pro public, Pro V2, Pro commercial, Terminal-Bench 2.0, 2.1, 3.0 and 4.0 are different tests. Numbers across versions are not comparable.
• Check who ran it and with what harness. Vendor-run results with a custom scaffold are not the same as a leaderboard run under fixed rules.
• Prefer private or held-out results. If a benchmark has a private split, that score is the better signal of generalisation.
• Look at cost and tokens. The Terminal-Bench 3.0 post reports GPT-5.6 Sol about 40% cheaper and using about 50% fewer tokens than Fable 5 for similar pass rates. For a team running agents all day, that matters more than a one-point lead.
• Watch for saturation. When the top entries are bunched within a few points near the ceiling, the benchmark has stopped separating models.

Run a small eval on your own code

The most reliable benchmark for your team is a few dozen of your own recently closed issues, replayed against the tools you are considering. Both benchmark harnesses are open source, so you can borrow their structure. The commands below are illustrative; check each project's README for current flags before running them.

Terminal-Bench 4.0 runs through Harbor; the dataset name comes from the official 4.0 post: harbor run -d terminal-bench/[email protected]

SWE-bench's harness grades a predictions file (one patch per task) in Docker: python -m swebench.harness.run_evaluation --dataset_name princeton-nlp/SWE-bench_Verified --predictions_path preds.jsonl --max_workers 8 --run_id my-eval

For a private eval, copy the pattern rather than the dataset: pick merged pull requests that came with tests, check out the parent commit, give the agent only the issue text, and score with the tests from the real fix. Track pass rate, cost and wall-clock time per task, and keep the set private so it never leaks into anyone's training data.

Our take

Stop treating SWE-bench Verified as a leaderboard; even OpenAI, which helped build it, no longer reports it. On SWE-Bench Pro, weight the private commercial results over the near-ceiling public V2 scores. Terminal-Bench 3.0 and 4.0 are currently the more useful public signal for autonomous agents, because they are hard enough to separate frontier models and are actively maintained, but always read them as model plus harness.

If you are choosing between Claude Code, Codex CLI or an open-weight model like Kimi K3.1 Pro, use public benchmarks to build a shortlist of two or three, then decide with a private eval on your own repository. That is the only score that measures the work you actually need done. Standards for trusting agents beyond benchmarks are still forming; see our look at agent certification.

Sources

• SWE-bench: original benchmark page and SWE-bench Verified, SWE-bench team
• SWE-bench: Can Language Models Resolve Real-world GitHub Issues?, Jimenez et al.
• Why we no longer evaluate SWE-bench Verified, OpenAI, February 23, 2026
• Separating signal from noise in coding evaluations, OpenAI, July 8, 2026
• SWE-Bench Pro paper and SWE-Bench Pro V2 post, Scale AI
• Terminal-Bench launch, 2.0 and Harbor, 2.1, 3.0, 4.0 and leaderboard integrity update, Terminal-Bench team