AI Model Wiki

    Skip to main content
    Knowledge

    MMLU (Massive Multitask Language Understanding)

    57-subject multiple-choice exam covering everything from US history to abstract algebra.

    What it measures

    Breadth of factual knowledge across humanities, STEM, social sciences, and professional domains.

    How it works

    15,908 multiple-choice questions across 57 subjects. The model is given a question and four answer choices and must pick the correct letter. Score is overall accuracy.

    Caveats

    • Heavily contaminated. Frontier models score 86-90% which suggests training-set overlap.
    • All questions are multiple-choice. Real-world tasks rarely come with four neat options.
    • Subject distribution is not uniform — some categories have many more questions and dominate the average.
    • Largely superseded by MMLU-Pro and GPQA, which were designed to resist Google-search shortcuts.

    Primary source

    hendrycks/test on GitHub

    Related benchmarks

    • MMLU-Pro — Harder MMLU with 10 answer choices, dropped trivial questions, and stronger reasoning requirements.
    • GPQA Diamond — Graduate-level science questions written to resist Google-search shortcuts.

    How we read this benchmark

    Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.