AI Model Wiki

    Skip to main content
    Latency

    Latency benchmarks (TTFT, throughput)

    Time-to-first-token and tokens-per-second — how quickly a model responds.

    What it measures

    User-facing latency. TTFT (time to first token) drives perceived snappiness; throughput drives total response time on long outputs.

    How it works

    Standardised prompts of varying length are sent to the model's API. TTFT measures wall-clock time from request to first streamed token; throughput is tokens emitted per second after streaming begins. Reported as median + p95.

    Caveats

    • Latency depends on provider's deployment, region, current load, and the model's batch policy — single readings are noisy.
    • Same model can have very different latency on different gateways (Together, Fireworks, Groq, the provider's own API).
    • Reasoning models that 'think' before answering have low throughput by design — comparing them to standard models is unfair without noting it.

    Primary source

    Artificial Analysis — Latency & throughput

    Related benchmarks

    How we read this benchmark

    Our Benchmark Methodology explains which benchmarks we cite on model pages, the caveats we apply, and why we never present any single benchmark as a verdict.