Multi-head attention is an extension of the Self-Attention mechanism in which the input is linearly projected into several separate query, key, and value spaces - called heads - and attention is computed independently in each. The outputs from all heads are concatenated and passed through a final linear layer. Running multiple heads in parallel lets the model learn to attend to different positional or semantic relationships simultaneously, rather than collapsing everything into a single weighted average.

Think of it like having several specialists read the same document at once: one focuses on who performed the action, another on when it happened, and a third on where. At the end, their separate annotations are merged into a richer understanding than any single reader could produce.

In the original Transformer paper, multi-head attention with h=8 heads is defined as:

import torch.nn as nn

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model=512, num_heads=8):
        super().__init__()
        self.attn = nn.MultiheadAttention(d_model, num_heads)

    def forward(self, query, key, value):
        output, weights = self.attn(query, key, value)
        return output  # shape: (seq_len, batch, d_model)

Each head uses a reduced dimension (d_model / num_heads) so total computation stays comparable to single-head attention at full dimension. Related concepts include Attention (the single-head foundation), the Transformer architecture that places multi-head attention at its core, Embedding layers that produce the input representations, and BERT and other pretrained models that stack many multi-head attention layers.

You will encounter multi-head attention in virtually every modern large language model and vision transformer, making it one of the most fundamental building blocks in contemporary deep learning research and deployment.