Prefix caching is an inference optimisation in which the key-value (KV) attention states computed for a fixed prompt prefix - such as a long system prompt or a shared document - are calculated once and reused across multiple subsequent requests, instead of being recomputed from scratch each time. Because the Transformer's Self-Attention mechanism must process every token in the Context Window on each forward pass, large static prefixes represent wasted work; caching them cuts latency and GPU cost proportionally to how much of the prompt is shared. Modern serving frameworks such as vLLM and SGLang implement prefix caching transparently using a radix-tree or hash-map keyed on token sequences.

Think of it like a photocopier that scans the cover page of a multi-page form once and keeps it in memory, so every time a new customer fills in only the last page you only need to scan the new part - the cover is already done.

Consider an API that always prepends the same 2,000-token legal disclaimer to every user query. Without prefix caching, those 2,000 tokens are processed on every call. With prefix caching enabled:

# vLLM - prefix caching is on by default; identical prefixes reuse KV cache
from vllm import LLM
llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.3", enable_prefix_caching=True)
responses = llm.generate(["<system>...2000-token-prefix...</system> User query 1",
                          "<system>...2000-token-prefix...</system> User query 2"])
# second call skips recomputing the shared prefix entirely

Prefix caching interacts with Quantization (both reduce inference cost but independently) and is especially valuable when combined with RAG pipelines that prepend the same retrieved documents to many queries. It also complements Prompt Engineering best practices of placing stable context at the start of a prompt to maximise cache hits.

You will encounter prefix caching in the documentation of any production LLM serving stack or API provider - OpenAI, Anthropic, and Google all offer some form of prompt caching that directly maps to this concept, often with per-token pricing discounts for cached portions.