The attention mechanism is a component in neural networks that allows the model to weigh the relevance of each part of its input when producing each part of its output. Rather than compressing the entire input into a single fixed vector, attention computes a score between every input position and every output position, then forms a weighted sum of input representations accordingly. In Transformer architectures, this is extended to self-attention, where each position in a sequence attends to every other position in the same sequence, letting the model capture long-range dependencies without relying on the sequential recurrence used in older RNN designs.

Think of it like highlighting a document before summarising it. You do not treat every sentence equally; you underline the parts most relevant to the question you are answering, then write your summary based mainly on those highlighted sections.

In a self-attention layer, three matrices — Query (Q), Key (K), and Value (V) — are computed from the input embeddings, and the output is calculated as:

Attention(Q, K, V) = softmax(Q * K^T / sqrt(d_k)) * V

The division by the square root of the key dimension d_k prevents the dot products from growing so large that the Softmax function saturates. Multi-head attention runs several such operations in parallel with different learned projections, allowing the model to attend to different aspects of the input simultaneously.

Attention is the defining building block of the Transformer architecture and underlies virtually every modern LLM, including BERT and its relatives. It connects closely to Embedding (the input representations that attention operates on), Context Window (attention scales quadratically with sequence length, which is why windows have practical limits), and Self-Attention as its most common instantiation.

You will encounter the attention mechanism in any paper or documentation discussing Transformer-based models, and its computational cost relative to sequence length is a central concern when evaluating inference efficiency.