What it measures
Genuine reasoning on hard biology, chemistry, and physics questions where the answer can't be found by web search.
How it works
448 questions in the Diamond subset, written by PhD-level domain experts and validated against open-internet shortcuts. Multiple choice. Score is accuracy.
Caveats
- Small dataset (~450 in Diamond) — single-percent-point swings are statistical noise.
- Expert authorship means questions are weighted toward what experts consider hard, not what end users actually need.
- Reasoning models with chain-of-thought outperform plain models by 10-20 points — score isn't only about underlying knowledge.
Primary source
Related benchmarks
- MMLU-Pro — Harder MMLU with 10 answer choices, dropped trivial questions, and stronger reasoning requirements.
- AIME (American Invitational Mathematics Examination) — 15-question math olympiad-track exam with integer answers in [0, 999].
- HumanEval — 164 Python coding problems with hidden unit tests.
How we read this benchmark
Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.