What it measures
Effective context window — how well a model retrieves, multi-hops, aggregates, and tracks variables across 4k–128k+ tokens.
How it works
13 synthetic tasks (needle-in-haystack, multi-key retrieval, variable tracking, common-words count, frequent-words count, QA). Each runs at multiple context lengths from 4k to 128k+. The 'effective' length is the longest at which accuracy stays above a threshold (typically 85-90%).
Caveats
- Synthetic tasks may not reflect what users actually do with long context (read a codebase, summarise a book, etc.).
- "Effective" cutoff depends on the chosen accuracy threshold.
- Most providers report only the headline context window. RULER often shows the usable window is 2-4x smaller.
Primary source
NVIDIA RULER (arXiv:2404.06654)
Related benchmarks
- LMSYS Chatbot Arena — Blind pairwise human voting on real chat completions, aggregated into an Elo-style ranking.
How we read this benchmark
Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.