Positional encoding is a mechanism that injects information about token order into the input representations of a transformer model, which would otherwise treat the sequence as an unordered set. Because transformer self-attention computes pairwise relationships between all tokens simultaneously — unlike recurrent networks that step through tokens sequentially — it has no inherent sense of which token came first. Positional encodings add a position-dependent signal to each token's embedding before it enters the attention layers, giving the model a way to distinguish "the dog bit the man" from "the man bit the dog."

Think of it like numbering seats in a lecture hall before handing out answer sheets. The students (tokens) sit in different seats (positions), and the seat number stamped on each sheet tells the grader where that answer came from — otherwise all sheets would look identical in terms of who answered what.

The original transformer paper used fixed sinusoidal encodings: for each position p and each dimension d of the embedding, the encoding alternates sine and cosine functions at geometrically spaced frequencies. Modern large language models favor Rotary Position Embeddings (RoPE), which encode relative positions by rotating query and key vectors before the dot product, giving better length generalization:

# Sinusoidal positional encoding (original Transformer)
import torch, math
def sinusoidal_encoding(seq_len, d_model):
    pe = torch.zeros(seq_len, d_model)
    pos = torch.arange(seq_len).unsqueeze(1).float()
    div = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(pos * div)
    pe[:, 1::2] = torch.cos(pos * div)
    return pe

The choice of positional encoding scheme directly affects a model's ability to generalize to sequences longer than those seen during training, which is why newer schemes like ALiBi and RoPE have largely replaced fixed sinusoidal encodings in production LLMs.

Positional encoding is inseparable from the Transformer and Self-Attention architecture — without it, attention has no positional signal. It also interacts with Context Window length, since some encoding schemes degrade or require extrapolation when sequences exceed the training length.

You will encounter positional encoding details most often when extending a model's context window, reading about long-context model architectures, or debugging generation failures on unusually long input sequences.