AI Model Wiki

    Skip to main content

    Benchmark Methodology

    Last updated: February 2026

    Benchmarks are useful and benchmarks are also misleading. Every leaderboard on this site comes with caveats, and we never present a benchmark score as a verdict.

    Benchmarks we cite

    • MMLU / MMLU-Pro — broad knowledge across 57 academic subjects
    • GPQA Diamond — graduate-level science questions designed to resist Google-search shortcuts
    • HumanEval / LiveCodeBench — code generation correctness on standalone functions
    • SWE-Bench / SWE-Bench Verified — real GitHub issues resolved end-to-end
    • Terminal-Bench / Agent-Bench — agentic task completion on shells and environments
    • AIME — competition math reasoning
    • LMSYS Chatbot Arena — human preference rankings from blind pairwise voting
    • Artificial Analysis — third-party cost and latency aggregator

    Each cited score links back to its source leaderboard or the developer's published number. Numbers older than 90 days are flagged.

    How we read them

    A benchmark measures a narrow task under specific conditions. The same model can rank #1 on one benchmark and mid-pack on another. We try to present at least three benchmarks per model spanning knowledge, reasoning, and coding so no single number frames the model.

    Caveats we apply

    • Contamination — frontier models are trained on huge web crawls; older benchmarks may overlap training data. We prefer benchmarks updated within the last year (e.g., LiveCodeBench, SWE-Bench Verified, GPQA Diamond).
    • Style bias on human-preference rankings — Chatbot Arena rewards engaging, conversational style. A model that gives shorter, more accurate answers can lose to one that pads.
    • Optimisation pressure — when developers know a benchmark is industry-defining, they fine-tune for it. The score becomes a measure of how much that benchmark was prioritised, not the model's general capability.
    • Inference settings — temperature, system prompts, retries, chain-of-thought, and tool use can all change the result by 5–15 points. We note the configuration when known.
    • Cherry-picked subsets — "coding benchmark" can mean HumanEval (easy) or SWE-Bench (hard). The figure changes by category. We name the exact benchmark.
    • Time decay — leaderboard positions change weekly. A benchmark cited six months ago is probably stale.

    What we never say

    • "Model X is the best AI" (without a use-case qualifier)
    • "X% better than competitor Y" (without a specific benchmark and date)
    • "Beats all other models" (no model beats all others on all benchmarks)

    Use-case-specific testing matters more

    Benchmark numbers are a starting filter. For any decision that matters — choosing a model for production, picking a vendor — we recommend running your own evals on prompts that look like your real workload. See our Evaluating & Benchmarking AI Models course for how.