Continuous batching is a serving strategy for large language models in which the inference engine dynamically adds new incoming requests to an in-progress batch as individual sequences finish generating, rather than waiting for every sequence in a fixed batch to complete before starting the next group. In standard static batching, a slow or long sequence blocks the entire batch from releasing; continuous batching eliminates that stall by treating each sequence's completion as an independent event and immediately backfilling the freed GPU slot with a waiting request. The result is substantially higher GPU utilization and lower average latency under variable load, which is why most production LLM serving systems including vLLM and TensorRT-LLM implement it by default.

Think of it like a restaurant where tables are cleared and reseated one at a time as diners finish, rather than waiting for every table in a section to empty before seating anyone new. The kitchen stays continuously busy regardless of how long each party lingers.

Consider a server handling requests of lengths 50, 200, and 800 tokens. With static batching, all three requests block until the 800-token sequence finishes. With continuous batching, the 50-token and 200-token responses are returned as soon as they complete, their GPU slots are reassigned to new arrivals, and only the long sequence holds its slot for the full duration. This can double or triple effective throughput on a busy endpoint.

Continuous batching is closely related to Inference optimization techniques, Quantization, both of which reduce the per-token cost of serving, the KV cache which continuous batching systems manage per sequence to enable dynamic scheduling, and Mixture of Experts architectures which alter the compute profile per token and interact with batching strategies.

You will encounter continuous batching in documentation for serving frameworks like vLLM, Triton Inference Server, and TGI, as well as in benchmarks comparing LLM serving throughput across hardware configurations.