Byte-Pair Encoding (BPE) is a data-compression-derived Tokenization algorithm that builds a subword vocabulary by iteratively merging the most frequent adjacent byte or character pairs in a training corpus. Starting from individual characters, each merge step replaces the top-frequency pair with a new token until the target vocabulary size is reached. The resulting vocabulary allows common words to be represented as single tokens while rare or unknown words are split into recognizable subword units.

Think of it like building a shorthand system for a language: you start writing every letter separately, then notice you always write "th" together, so you create a new symbol for it—and keep doing that for the most common combos until your shorthand is efficient.

GPT-2, GPT-3, GPT-4, and many other Transformer models use BPE (or its byte-level variant). The word "unhappiness" might be tokenized as ["un", "happiness"] or ["un", "happy", "ness"], keeping vocabulary size manageable while handling morphology gracefully. The vocabulary is typically 32k–100k tokens.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")  # GPT-4 encoding
tokens = enc.encode("unhappiness")
print(tokens)           # e.g. [1904, 42799]
print(len(tokens))      # 2

BPE sits within the broader Tokenization landscape alongside alternatives like WordPiece (used by BERT) and Unigram Language Model tokenizers. It directly affects Context Window usage, since a longer average token length means fewer real words fit within a model's token limit. Fine-tuning a model trained with a specific BPE vocabulary on a new language often requires vocabulary extension because the base merges may undersegment that language's morphemes.

You will encounter BPE when configuring tokenizers in libraries like Hugging Face Transformers, when estimating prompt costs in LLM APIs, and in discussions of how Pretraining vocabulary choices affect downstream task performance.