AI Model Wiki

    Skip to main content
    Coding

    HumanEval

    164 Python coding problems with hidden unit tests.

    What it measures

    Ability to generate a correct standalone Python function from a docstring and signature.

    How it works

    Model is given a function signature + docstring. It outputs the function body. Hidden test cases evaluate correctness. Score is pass@1 (single attempt) or pass@k.

    Caveats

    • Largely solved. Frontier models score 88-95%. Useful for low-end model differentiation only.
    • Standalone functions are nothing like real codebases. No multi-file context, no API design, no debugging existing code.
    • Contamination is plausible — the dataset is small and old.
    • Python only.

    Primary source

    openai/human-eval on GitHub

    Related benchmarks

    • LiveCodeBench — Continuously-updated competitive-programming problems from LeetCode, AtCoder, and Codeforces.
    • SWE-Bench — Real GitHub issues from 12 popular Python repos that the model must resolve end-to-end.
    • SWE-Bench Verified — 500-instance cleaned subset of SWE-Bench with verified test reliability.

    How we read this benchmark

    Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.