Flash Attention is a memory-efficient algorithm for computing the scaled dot-product Attention operation that is the core of Transformer models. Standard attention requires materialising the full N x N attention matrix in GPU high-bandwidth memory (HBM), which becomes prohibitively large as sequence length N grows — memory usage scales quadratically. Flash Attention avoids this by tiling the computation into blocks that fit in the GPU's faster on-chip SRAM, computing exact attention results incrementally without ever writing the full matrix to HBM. The algorithm produces numerically identical output to standard attention while achieving 2-4x wallclock speedups and O(N) rather than O(N squared) memory usage.

Think of it like reading a very long book by keeping only one chapter open on your desk at a time rather than photocopying every page and spreading them across the floor. You get the same understanding of the whole book but use a fraction of the desk space and avoid the slow trip to the photocopier for each page.

The practical impact is that Flash Attention allows training and inference with dramatically longer Context Windows on the same hardware. A model that previously could only handle 2K tokens due to memory constraints can be extended to 8K or 32K tokens with Flash Attention v2 without changing model weights or architecture. It is implemented in libraries such as xFormers, and is enabled by default in modern versions of PyTorch via F.scaled_dot_product_attention.

Flash Attention is an optimisation of the core Attention and Self-Attention mechanism inside Transformer architectures. It directly enables longer Context Windows, which matter for RAG (larger retrieved documents fit in a single pass) and Chain of Thought reasoning (longer intermediate steps are possible). The technique is architecture-agnostic and applies equally to LLMs, Multimodal models, and Diffusion transformers.

You will encounter Flash Attention in model training papers, in hardware selection discussions (it is optimised for NVIDIA A100/H100 tensor cores), and in open-source inference frameworks where enabling the flag is a standard performance tuning step.