Long context refers to the ability of a language model to process and reason over input sequences that span tens of thousands to millions of tokens within a single Context Window. Early Transformer models were limited to 512 or 2048 tokens; modern architectures using techniques such as rotary positional embeddings, sliding window attention, and sparse attention patterns have extended practical limits to 128K, 200K, or beyond, enabling whole codebases, lengthy legal documents, or multi-hour transcripts to be processed in one pass.

Think of it like the difference between reading a novel with a bookmark and reading the entire book at once. With a short context, the model must summarise and chunk; with a long context, it can reference any passage directly without information loss from compression.

A long-context model can answer questions that require synthesising information from page 2 and page 300 of a document simultaneously. This is particularly valuable for RAG pipelines where entire retrieved documents, rather than truncated excerpts, can be included. It also reduces the need for complex chunking strategies that degrade retrieval accuracy.

Long context capability depends on how the Self-Attention mechanism scales with sequence length. Vanilla Attention has quadratic time and memory complexity in sequence length, which is why architectural modifications are needed. Models like Claude 3 with 200K context and Gemini 1.5 with 1M context use modified Attention that reduces this cost. Long context also interacts with Hallucination risk: models sometimes fail to retrieve information from the middle of very long inputs, a phenomenon called the lost-in-the-middle problem. Chain of Thought and careful Prompt Engineering help mitigate this. Embedding-based RAG becomes less necessary as Context Windows grow, though it remains cost-effective for large corpora.

You will encounter long-context models when processing legal contracts, large codebases, research papers, or any task requiring cross-document reasoning.