MMLU, or Massive Multitask Language Understanding, is a benchmark designed to evaluate an AI model's breadth of world knowledge and reasoning by testing it across 57 academic and professional subjects in a multiple-choice format. Subjects range from elementary mathematics and US history to college-level chemistry, professional law, and clinical medicine. Each question has four answer options, and chance performance is 25%; human expert performance averages around 89%, providing an upper reference point. Because MMLU covers such diverse domains, a model's aggregate score is widely treated as a proxy for general knowledge and reasoning capability.

Think of it like a general-knowledge trivia marathon that spans every subject taught from middle school through graduate school: a contestant who scores well across all categories has demonstrated genuinely broad comprehension rather than deep expertise in just one field.

A standard MMLU evaluation feeds each question as a zero-shot or few-shot prompt to the model and records which answer letter it assigns the highest probability:

Question: A 45-year-old patient presents with chest pain...
(A) Aortic dissection  (B) GERD  (C) Pulmonary embolism  (D) Costochondritis
Model log-probs: A=0.41, B=0.09, C=0.38, D=0.12  -> predicts A

GPT-4 scored approximately 86% at launch. MMLU connects directly to concepts like Perplexity (a related measure of model uncertainty), Zero-Shot and Chain of Thought prompting strategies used to elicit answers, and LLM benchmarking practice generally.

You will encounter MMLU scores in almost every frontier model release announcement and in research papers comparing Pretraining data mixtures, Fine-tuning strategies, or the effect of scale on general knowledge.