Benchmarks & Deep Dives · Free

SWE-bench

SWE-bench is an open-source benchmark that tests whether an AI system can resolve real GitHub issues. Each task gives the model an issue and a repository checked out at the right commit; the fix counts only if the repository's own tests pass afterwards. The original set (October 2023) has 2,294 tasks from 12 popular Python repositories, and the family now includes SWE-bench Verified (a 500-task human-validated subset built with OpenAI), Lite, Multilingual and Multimodal. It is maintained by the SWE-bench team behind SWE-agent and mini-SWE-agent, and is free to run with its Docker-based evaluation harness.