Benchmark Methodology
Last updated: February 2026
Benchmarks are useful and benchmarks are also misleading. Every leaderboard on this site comes with caveats, and we never present a benchmark score as a verdict.
Benchmarks we cite
- MMLU / MMLU-Pro — broad knowledge across 57 academic subjects
- GPQA Diamond — graduate-level science questions designed to resist Google-search shortcuts
- HumanEval / LiveCodeBench — code generation correctness on standalone functions
- SWE-Bench / SWE-Bench Verified — real GitHub issues resolved end-to-end
- Terminal-Bench / Agent-Bench — agentic task completion on shells and environments
- AIME — competition math reasoning
- LMSYS Chatbot Arena — human preference rankings from blind pairwise voting
- Artificial Analysis — third-party cost and latency aggregator
Each cited score links back to its source leaderboard or the developer's published number. Numbers older than 90 days are flagged.
How we read them
A benchmark measures a narrow task under specific conditions. The same model can rank #1 on one benchmark and mid-pack on another. We try to present at least three benchmarks per model spanning knowledge, reasoning, and coding so no single number frames the model.
Caveats we apply
- Contamination — frontier models are trained on huge web crawls; older benchmarks may overlap training data. We prefer benchmarks updated within the last year (e.g., LiveCodeBench, SWE-Bench Verified, GPQA Diamond).
- Style bias on human-preference rankings — Chatbot Arena rewards engaging, conversational style. A model that gives shorter, more accurate answers can lose to one that pads.
- Optimisation pressure — when developers know a benchmark is industry-defining, they fine-tune for it. The score becomes a measure of how much that benchmark was prioritised, not the model's general capability.
- Inference settings — temperature, system prompts, retries, chain-of-thought, and tool use can all change the result by 5–15 points. We note the configuration when known.
- Cherry-picked subsets — "coding benchmark" can mean HumanEval (easy) or SWE-Bench (hard). The figure changes by category. We name the exact benchmark.
- Time decay — leaderboard positions change weekly. A benchmark cited six months ago is probably stale.
What we never say
- "Model X is the best AI" (without a use-case qualifier)
- "X% better than competitor Y" (without a specific benchmark and date)
- "Beats all other models" (no model beats all others on all benchmarks)
Use-case-specific testing matters more
Benchmark numbers are a starting filter. For any decision that matters — choosing a model for production, picking a vendor — we recommend running your own evals on prompts that look like your real workload. See our Evaluating & Benchmarking AI Models course for how.