What it measures
Human preference under realistic chat use — answer quality, helpfulness, and style as perceived by users.
How it works
Anonymous users send a prompt and see two model responses side-by-side without knowing which model produced which. They vote for the better answer (or a tie). Votes update an Elo-style rating; the leaderboard publishes ratings updated weekly.
Caveats
- Style bias: engaging, conversational answers beat shorter accurate ones. Models tuned for friendliness can outrank stronger reasoning models.
- Prompt distribution is whatever users type — heavily skewed toward chat, coding, and easy questions.
- Vote integrity: voters can theoretically be sloppy or partisan. The Arena does noise filtering but it's imperfect.
- Same model, different system prompt: different Elo. Provider versions matter.
Primary source
LMSYS Chatbot Arena Leaderboard
Related benchmarks
- MMLU (Massive Multitask Language Understanding) — 57-subject multiple-choice exam covering everything from US history to abstract algebra.
- GPQA Diamond — Graduate-level science questions written to resist Google-search shortcuts.
How we read this benchmark
Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.