The Transformer is a neural network architecture introduced in the 2017 paper "Attention Is All You Need" that replaced recurrent and convolutional designs for sequence modelling. Its core innovation is self-attention: rather than processing tokens one at a time, every token in the input simultaneously computes weighted relationships to every other token, allowing the network to capture long-range dependencies in a single layer. The architecture stacks many such Attention layers, interleaved with feed-forward blocks and layer normalisation, and can process entire sequences in parallel on modern hardware—making it dramatically faster to train than its predecessors.

Think of it like a roundtable meeting where every participant listens to every other participant at once and weights their input by relevance, instead of a telephone chain where information passes one person at a time. That simultaneous, weighted listening is Self-Attention.

import torch
from torch.nn import TransformerEncoderLayer
layer = TransformerEncoderLayer(d_model=512, nhead=8, batch_first=True)
x = torch.rand(32, 64, 512)  # batch=32, seq_len=64, d_model=512
out = layer(x)               # shape unchanged: (32, 64, 512)

Each of the eight attention heads learns a different relational pattern—syntax, coreference, topic—simultaneously.

The Transformer is the backbone of virtually every modern LLM, including GPT, Claude, and Gemini, and has expanded into vision (Multimodal models) and audio. Its design directly enables BERT (encoder-only), GPT (decoder-only), and T5 (encoder-decoder) variants.

You'll encounter the Transformer architecture in every research paper on language, vision, or audio AI published after 2018, and in any framework code for loading, fine-tuning, or deploying state-of-the-art models.