Throughput, in the context of AI inference, is the number of tokens (or complete requests) a deployed model can process per unit of time — typically measured as tokens per second or requests per minute. It is a system-level metric that captures the combined effect of hardware speed, batching strategy, model size, and serving software efficiency, and it stands in deliberate tension with latency: optimisations that maximise throughput (larger batches, longer queues) often increase the time any single request waits for a response.

Think of it like a highway toll plaza. Adding more lanes (GPUs) and grouping cars into convoys (batching) raises the total number of vehicles processed per hour — throughput — even if individual drivers spend slightly longer queued before they reach the booth.

# Benchmark throughput with vLLM
python -m vllm.entrypoints.benchmark_throughput   --model meta-llama/Llama-3-8B-Instruct   --num-prompts 1000   --input-len 512   --output-len 128
# Reports: Throughput: 2314.7 tokens/s

Raising batch size is the single most effective lever: a batch of 32 requests on one A100 can achieve 5–10× the throughput of serving them one at a time, at the cost of higher per-request latency.

Throughput is tightly coupled to Quantization — lower-precision weights fit larger activation batches in GPU memory — and to Inference optimisation techniques like speculative decoding, continuous batching, and paged attention (used by vLLM). It differs from Parameter count or Perplexity, which measure model capability rather than serving efficiency. Mixture of Experts architectures can increase effective model size while keeping per-token FLOPs — and therefore throughput — relatively high.

You will encounter throughput as a key metric when sizing GPU clusters for production deployments, comparing serving frameworks, or calculating cost-per-query for large-scale LLM APIs.