A KV Cache (Key-Value Cache) is a memory optimisation used during autoregressive text generation in transformer models. During the forward pass, every token's attention layer computes a key vector and a value vector; once computed for a given position, those vectors do not change as subsequent tokens are generated. The KV Cache stores these computed keys and values so that each new decoding step can attend to the full history without recomputing it, reducing redundant work that would otherwise scale quadratically with sequence length.

Think of it like a notepad where a reader jots down the main points of every sentence in a document as they read. When they need to answer a question about an earlier sentence, they consult their notes rather than re-reading the whole document from the start.

In practice, the cache grows by one key-value pair per layer per new token. For a model with 96 layers, a context of 4,096 tokens, and a hidden dimension of 128 per head, the cache can occupy several gigabytes of GPU memory. This is why providers offering long-context inference (100K+ tokens) must manage KV Cache size carefully, sometimes evicting older entries or using quantised 8-bit key/value storage to stay within VRAM limits. The tradeoff appears directly in API pricing: Anthropic's prompt-caching feature charges a reduced rate for tokens whose KV representations are already stored.

KV Caching is central to Inference efficiency and is tightly coupled with Attention and Self-Attention mechanics inside the Transformer architecture. It also intersects with Context Window management, since a larger context window demands proportionally more cache memory. Quantization of the cache itself is an active research area to reduce that memory footprint.

You will encounter KV Cache as a concept when optimising inference throughput, reading about speculative decoding, or diagnosing out-of-memory errors on long-context generation tasks.