Benchmark Interpreter
Tell us what you'll use the model for. We'll rank which benchmarks are worth paying attention to, which to ignore, and what to test for yourself before committing.
Benchmarks worth your attention
- SWE-Bench VerifiedMatters most
Resolves real GitHub issues end-to-end on cleaned tests — the closest proxy to engineering work.
- LiveCodeBenchMatters most
Contamination-resistant algorithmic problems; filter to dates after the model's cutoff.
- HumanEvalMinor signal
Frontier models score 88–95%; useful for differentiating smaller models only.
- Terminal-BenchWorth checking
Matters if you're running an agent that issues shell commands.
Deprioritise these
These benchmarks are often cited but don't predict your use case well.
- LMSYS Chatbot Arena — Style bias means strong reasoners can lose to friendly-sounding ones.
What to test yourself
Benchmark scores are a filter, not a verdict. Before you commit to a model:
- Run the candidate models on 20-50 representative prompts from your actual workload.
- Compare outputs blind — random order, no labels — so you don't anchor on the brand.
- Re-test on every major model version change; rankings change meaningfully between releases.
- For agent / tool-use cases, eval the whole stack (model + harness), not just the model.
- See the Evaluating & Benchmarking AI Models course for how to run your own evals.