AI Model Wiki

    Skip to main content
    Long Context

    RULER

    Tests how much of a claimed long context window is actually usable.

    What it measures

    Effective context window — how well a model retrieves, multi-hops, aggregates, and tracks variables across 4k–128k+ tokens.

    How it works

    13 synthetic tasks (needle-in-haystack, multi-key retrieval, variable tracking, common-words count, frequent-words count, QA). Each runs at multiple context lengths from 4k to 128k+. The 'effective' length is the longest at which accuracy stays above a threshold (typically 85-90%).

    Caveats

    • Synthetic tasks may not reflect what users actually do with long context (read a codebase, summarise a book, etc.).
    • "Effective" cutoff depends on the chosen accuracy threshold.
    • Most providers report only the headline context window. RULER often shows the usable window is 2-4x smaller.

    Primary source

    NVIDIA RULER (arXiv:2404.06654)

    Related benchmarks

    • LMSYS Chatbot Arena — Blind pairwise human voting on real chat completions, aggregated into an Elo-style ranking.

    How we read this benchmark

    Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.