AI Model Wiki

    Skip to main content
    Knowledge

    MMLU-Pro

    Harder MMLU with 10 answer choices, dropped trivial questions, and stronger reasoning requirements.

    What it measures

    Same breadth as MMLU but with substantially less head-room for frontier models.

    How it works

    12,032 questions distilled from MMLU plus additional STEM problems. Increased from 4 to 10 answer choices and removed questions answered correctly by smaller models.

    Caveats

    • Still multiple-choice — not a generative test.
    • Frontier-model scores already approaching ceiling.
    • Score is sensitive to the prompt format used by the evaluator.

    Primary source

    TIGER-Lab/MMLU-Pro on Hugging Face

    Related benchmarks

    How we read this benchmark

    Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.