AI Model Wiki

    Skip to main content
    Reasoning

    GPQA Diamond

    Graduate-level science questions written to resist Google-search shortcuts.

    What it measures

    Genuine reasoning on hard biology, chemistry, and physics questions where the answer can't be found by web search.

    How it works

    448 questions in the Diamond subset, written by PhD-level domain experts and validated against open-internet shortcuts. Multiple choice. Score is accuracy.

    Caveats

    • Small dataset (~450 in Diamond) — single-percent-point swings are statistical noise.
    • Expert authorship means questions are weighted toward what experts consider hard, not what end users actually need.
    • Reasoning models with chain-of-thought outperform plain models by 10-20 points — score isn't only about underlying knowledge.

    Primary source

    GPQA paper (arXiv:2311.12022)

    Related benchmarks

    How we read this benchmark

    Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.