What it measures
Same as SWE-Bench but every instance has been hand-validated to ensure the tests actually pass on the gold patch and fail on the broken baseline.
How it works
Identical to SWE-Bench, restricted to the 500-instance Verified subset. OpenAI's resolve-rate methodology.
Caveats
- Smaller dataset means single-instance flukes affect the percentage.
- Agent scaffolding (Aider, OpenHands, SWE-Agent, etc.) materially changes the score — "Model X scores 50% on SWE-Bench Verified" is incomplete without naming the agent.
- Python only.
Primary source
OpenAI — Introducing SWE-Bench Verified
Related benchmarks
- SWE-Bench — Real GitHub issues from 12 popular Python repos that the model must resolve end-to-end.
- AgentBench — Multi-environment benchmark for LLM-driven agents — OS, DB, web, games, scientific.
- Terminal-Bench — Agentic tasks completed in a real shell — installing, debugging, running multi-step workflows.
How we read this benchmark
Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.