What it measures
Same breadth as MMLU but with substantially less head-room for frontier models.
How it works
12,032 questions distilled from MMLU plus additional STEM problems. Increased from 4 to 10 answer choices and removed questions answered correctly by smaller models.
Caveats
- Still multiple-choice — not a generative test.
- Frontier-model scores already approaching ceiling.
- Score is sensitive to the prompt format used by the evaluator.
Primary source
TIGER-Lab/MMLU-Pro on Hugging Face
Related benchmarks
- MMLU (Massive Multitask Language Understanding) — 57-subject multiple-choice exam covering everything from US history to abstract algebra.
- GPQA Diamond — Graduate-level science questions written to resist Google-search shortcuts.
How we read this benchmark
Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.