A tokenizer converts raw text into a sequence of tokens—discrete units such as words, subwords, or individual characters—that a neural network can process numerically. Modern tokenizers used in large language models (LLMs) typically apply algorithms like Byte Pair Encoding (BPE) or WordPiece, which break uncommon words into smaller subword fragments while keeping frequent words intact. The vocabulary built during tokenization maps each unique token to an integer ID, turning "unbelievable" into something like [317, 8997] rather than a single unit.

Think of it like a chef's mise en place: before cooking, every ingredient is cleaned, chopped, and placed in labeled bowls so the recipe can proceed without interruption. The tokenizer is that prep step—converting messy, variable-length human text into uniform pieces the model knows how to handle.

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokens = tokenizer("Hello, world!")
print(tokens["input_ids"])  # [15496, 11, 995, 0]

The same sentence tokenizes differently depending on the model's vocabulary, which is why token counts—and therefore costs—vary across providers.

Tokenization sits at the boundary between Tokenization strategy and LLM architecture; the vocabulary size is itself a Hyperparameter that affects both model capacity and Inference speed. Tokenization choices also influence Embedding quality because each token ID is mapped to a learned vector.

You'll encounter tokenizers whenever you estimate API costs (most providers charge per token), debug unexpected model behaviour on rare words or code symbols, or build a Prompt Engineering pipeline where precise token budgeting matters.