What it measures
Engineering ability: reading a real codebase, understanding an issue, producing a working patch that passes the maintainer's tests.
How it works
2,294 instances of (repo, issue, gold patch, tests). The model is given the issue text and repo at the parent commit. It outputs a patch; the project's existing tests + a regression test from the gold patch must pass. Score is resolve rate.
Caveats
- Many of the 2,294 instances have wrong or under-specified tests. Use SWE-Bench Verified instead for a cleaned subset.
- Setup is heavy — model must navigate a real repo. Score includes a lot of agent scaffolding, not just model capability.
- Python only. Repos skew toward Django, sympy, scikit-learn, etc.
Primary source
Related benchmarks
- SWE-Bench Verified — 500-instance cleaned subset of SWE-Bench with verified test reliability.
- LiveCodeBench — Continuously-updated competitive-programming problems from LeetCode, AtCoder, and Codeforces.
- AgentBench — Multi-environment benchmark for LLM-driven agents — OS, DB, web, games, scientific.
How we read this benchmark
Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.