What it measures
Ability to generate a correct standalone Python function from a docstring and signature.
How it works
Model is given a function signature + docstring. It outputs the function body. Hidden test cases evaluate correctness. Score is pass@1 (single attempt) or pass@k.
Caveats
- Largely solved. Frontier models score 88-95%. Useful for low-end model differentiation only.
- Standalone functions are nothing like real codebases. No multi-file context, no API design, no debugging existing code.
- Contamination is plausible — the dataset is small and old.
- Python only.
Primary source
Related benchmarks
- LiveCodeBench — Continuously-updated competitive-programming problems from LeetCode, AtCoder, and Codeforces.
- SWE-Bench — Real GitHub issues from 12 popular Python repos that the model must resolve end-to-end.
- SWE-Bench Verified — 500-instance cleaned subset of SWE-Bench with verified test reliability.
How we read this benchmark
Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.