Rotary Position Embedding (RoPE) is a positional encoding scheme for transformer models that represents each token's position by rotating its query and key vectors in a high-dimensional space. Instead of adding a fixed position vector to an embedding at the start, RoPE multiplies queries and keys by rotation matrices whose angle is a function of absolute position — the result is that the dot-product attention score depends only on the relative distance between two tokens, not their absolute indices. This makes RoPE naturally suited for generalising to sequence lengths longer than those seen during training.

Think of it like using a clock face rather than a numbered list to describe time. Two events can be described by where each hand points, and the gap between them is immediately readable from the angle between the hands — you don't need to know the absolute "slot number" of either event to compute the difference.

import torch

def rope_rotate(x, cos, sin):
    # x: [batch, heads, seq, dim], cos/sin: [seq, dim/2]
    x1, x2 = x[..., ::2], x[..., 1::2]
    return torch.cat([x1 * cos - x2 * sin,
                      x1 * sin + x2 * cos], dim=-1)

RoPE is closely tied to Self-Attention, Transformer, and Embedding design; it directly influences how a model handles Context Window extension. Many long-context techniques such as YaRN and LongRoPE build on top of it.

You will encounter RoPE when reading the architecture papers for Llama, Mistral, Qwen, or any model that advertises strong long-context performance, as it has become the de-facto positional encoding in open-weight LLMs.