What it measures
Generalisation across heterogeneous agent tasks rather than a single task family.
How it works
8 environments (OS, DB, knowledge graph, digital card game, lateral thinking puzzles, household, web shopping, web browsing). Each environment has its own success metric; final score is normalised average.
Caveats
- Average across very different tasks is hard to interpret. Strong on web tasks, weak on OS, etc.
- Some environments are gameable with simple heuristics; AgentBench score does not perfectly correlate with real-world agent quality.
Primary source
AgentBench paper (arXiv:2308.03688)
Related benchmarks
- Terminal-Bench — Agentic tasks completed in a real shell — installing, debugging, running multi-step workflows.
- SWE-Bench Verified — 500-instance cleaned subset of SWE-Bench with verified test reliability.
How we read this benchmark
Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.