What it measures
End-to-end shell competence: file manipulation, package installation, multi-step workflows, error recovery.
How it works
Each task gives the agent a goal and a containerised Linux environment. The agent issues shell commands; success is graded by a test harness. Score is task-completion rate.
Caveats
- Heavy dependence on agent harness — same model can score 30% or 60% depending on tool wrapper.
- Tasks skew toward Linux + Python. Limited coverage of other ecosystems.
- Currently a small dataset; absolute scores will move as the benchmark grows.
Primary source
Terminal-Bench (terminal-bench.org)
Related benchmarks
- AgentBench — Multi-environment benchmark for LLM-driven agents — OS, DB, web, games, scientific.
- SWE-Bench Verified — 500-instance cleaned subset of SWE-Bench with verified test reliability.
How we read this benchmark
Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.