vLLM is an open-source inference engine for large language models that achieves high throughput and low latency through two key innovations: PagedAttention and continuous batching. PagedAttention manages the KV cache — the memory that stores attention states for each token — in non-contiguous memory blocks analogous to virtual memory paging in operating systems, dramatically reducing fragmentation and enabling far more concurrent requests than naive implementations. Continuous batching means that as soon as one sequence in a batch finishes generating, a new request immediately takes its slot, keeping GPU utilisation near 100 % rather than waiting for an entire batch to complete.

Think of it like a busy restaurant kitchen that never lets a burner sit idle: the moment one dish leaves the pass, the chef immediately starts the next order on that ring rather than waiting for the whole table's tickets to clear before picking up fresh work. The result is the same number of burners serving far more diners per hour.

Deploying a model with vLLM takes a single command:

python -m vllm.entrypoints.openai.api_server   --model mistralai/Mistral-7B-Instruct-v0.2   --dtype bfloat16   --max-model-len 8192

This starts an OpenAI-compatible HTTP server that any existing client library can call immediately, with vLLM handling batching and memory management transparently.

vLLM is closely related to Inference optimisation, Quantization (which is often applied before deploying via vLLM to reduce memory footprint), LLM serving architecture, and Context Window management since PagedAttention directly addresses the cost of long contexts. Knowledge Distillation produces smaller models that vLLM can serve even more efficiently.

You will encounter vLLM when deploying open-source language models in production, particularly when throughput requirements exceed what Hugging Face's default generate() loop can sustain.