Self-attention is the mechanism at the heart of transformer architectures that allows every token in a sequence to directly query, and be influenced by, every other token in the same sequence. For each position, the mechanism computes three vectors — query, key, and value — and produces an output that is a weighted sum of all value vectors, where the weights are determined by how well that position's query matches every other position's key. The result is that each token's representation is updated with context drawn from the entire sequence in a single pass, capturing both short- and long-range dependencies simultaneously.

Think of it like a conference call where every participant can hear everyone else speak at once and decide, for themselves, how much weight to give each voice before forming their own response — rather than passing notes down a chain where information degrades with distance.

import torch, math

def self_attention(Q, K, V):
    d_k = Q.size(-1)
    scores = (Q @ K.transpose(-2, -1)) / math.sqrt(d_k)
    weights = torch.softmax(scores, dim=-1)   # see Softmax
    return weights @ V

Self-attention is the defining operation of the Transformer; it relies on Softmax to normalise scores and Embedding vectors as its inputs. The number of tokens it can attend to at once is bounded by the model's Context Window, and the broader Attention article covers the family of mechanisms it belongs to.

You will encounter self-attention in virtually every modern NLP and vision model — from BERT and GPT to vision transformers and multimodal systems — because it replaced recurrent layers as the primary context-aggregation mechanism.