Sliding window attention is a variant of the standard Attention mechanism that restricts each token to attending only to a fixed-size neighborhood of nearby tokens rather than every other token in the sequence. Because full self-attention scales quadratically with sequence length, replacing it with local windows reduces memory and compute to a near-linear relationship, making it practical to process documents, audio, or genomic sequences that would otherwise overflow GPU memory. Longformer and BigBird popularized this design, often combining local windows with a small number of global tokens that can attend everywhere to capture long-range dependencies.

Think of it like reading a very long scroll by holding a magnifying lens that covers only a few centimeters at a time and sliding it from left to right. You read every word with full attention within the lens, but you never try to hold the entire scroll in view simultaneously—a few special bookmarks at fixed positions give you a rough map of the whole document.

In practice, a model configured with a window size of 512 on a 16 000-token document attends, for each token, only to the 256 tokens before and after it. A configuration might look like attention_window = [512] * num_layers in Longformer's config, and the model handles the rest automatically during both training and Inference.

Sliding window attention is closely related to Self-Attention and the broader Transformer architecture; it is also a practical complement to techniques like Context Window extension and Quantization when deploying long-context models efficiently. You will encounter it whenever a research paper or model card discusses processing documents or code repositories that exceed the context limits of standard transformers.