AI Model Wiki

    Skip to main content
    Coding

    SWE-Bench

    Real GitHub issues from 12 popular Python repos that the model must resolve end-to-end.

    What it measures

    Engineering ability: reading a real codebase, understanding an issue, producing a working patch that passes the maintainer's tests.

    How it works

    2,294 instances of (repo, issue, gold patch, tests). The model is given the issue text and repo at the parent commit. It outputs a patch; the project's existing tests + a regression test from the gold patch must pass. Score is resolve rate.

    Caveats

    • Many of the 2,294 instances have wrong or under-specified tests. Use SWE-Bench Verified instead for a cleaned subset.
    • Setup is heavy — model must navigate a real repo. Score includes a lot of agent scaffolding, not just model capability.
    • Python only. Repos skew toward Django, sympy, scikit-learn, etc.

    Primary source

    SWE-Bench leaderboard

    Related benchmarks

    • SWE-Bench Verified — 500-instance cleaned subset of SWE-Bench with verified test reliability.
    • LiveCodeBench — Continuously-updated competitive-programming problems from LeetCode, AtCoder, and Codeforces.
    • AgentBench — Multi-environment benchmark for LLM-driven agents — OS, DB, web, games, scientific.

    How we read this benchmark

    Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.