AI Model Wiki

    Skip to main content

    Benchmark Interpreter

    Tell us what you'll use the model for. We'll rank which benchmarks are worth paying attention to, which to ignore, and what to test for yourself before committing.

    Benchmarks worth your attention

    • SWE-Bench Verified
      Matters most

      Resolves real GitHub issues end-to-end on cleaned tests — the closest proxy to engineering work.

    • LiveCodeBench
      Matters most

      Contamination-resistant algorithmic problems; filter to dates after the model's cutoff.

    • HumanEval
      Minor signal

      Frontier models score 88–95%; useful for differentiating smaller models only.

    • Terminal-Bench
      Worth checking

      Matters if you're running an agent that issues shell commands.

    Deprioritise these

    These benchmarks are often cited but don't predict your use case well.

    What to test yourself

    Benchmark scores are a filter, not a verdict. Before you commit to a model:

    • Run the candidate models on 20-50 representative prompts from your actual workload.
    • Compare outputs blind — random order, no labels — so you don't anchor on the brand.
    • Re-test on every major model version change; rankings change meaningfully between releases.
    • For agent / tool-use cases, eval the whole stack (model + harness), not just the model.
    • See the Evaluating & Benchmarking AI Models course for how to run your own evals.