AI Model Wiki

    Skip to main content
    15 min read
    Lesson 2 of 6

    Standard Benchmarks: MMLU, HumanEval & Beyond

    From the course Evaluating & Benchmarking AI Models

    Benchmarks provide standardized measurements that enable fair comparison between models. Understanding what each benchmark measures helps you interpret leaderboard results.

    Knowledge & Reasoning Benchmarks

    BenchmarkWhat It MeasuresFormatNotes
    MMLU57 academic subjectsMultiple choiceMost widely cited; some contamination concerns
    MMLU-ProHarder version of MMLU10-option MCReduces random guessing advantage
    ARCGrade-school scienceMultiple choiceEasy/Challenge splits
    HellaSwagCommonsense reasoningSentence completionAdversarially constructed
    WinograndePronoun resolutionBinary choiceTests world knowledge
    GSM8KGrade-school mathFree-formChain-of-thought helps dramatically
    MATHCompetition mathematicsFree-formVery challenging; tests formal reasoning

    Coding Benchmarks

    BenchmarkWhat It MeasuresFormat
    HumanEvalPython function generationCode completion
    HumanEval+Extended HumanEval with more testsCode completion
    MBPPMostly basic Python problemsCode generation
    SWE-benchReal GitHub issue resolutionEnd-to-end coding
    LiveCodeBenchFresh competitive programmingCode generation

    Multimodal Benchmarks

    • MMMU: Multi-discipline multimodal understanding across 30 subjects
    • MathVista: Mathematical reasoning with visual inputs (charts, diagrams)
    • RealWorldQA: Real-world image understanding

    Interpreting Benchmark Results

    • Absolute scores matter less than relative performance on tasks relevant to your use case
    • A model scoring 90% on MMLU but 40% on MATH has very different capabilities than one scoring 80% on both
    • Always check the evaluation methodology — 0-shot vs. 5-shot prompting dramatically changes scores
    • Date matters: Newer benchmarks are less likely to be contaminated in training data

    Key Takeaways

    • •MMLU, HumanEval, and GSM8K are the most commonly cited benchmarks for LLM comparison
    • •No single benchmark captures overall model quality — use task-relevant benchmarks
    • •Evaluation methodology (0-shot vs. few-shot) significantly affects benchmark scores
    • •Newer benchmarks like SWE-bench and LiveCodeBench are harder to game through contamination
    0 of 6 lessons marked complete

    Progress is kept in this browser only.