Tokenization is the preprocessing step that converts raw text into a sequence of integer IDs — tokens — that a neural network can ingest. Rather than operating on characters or dictionary words, most modern language models use subword algorithms: Byte-Pair Encoding (BPE), used by GPT-family models, iteratively merges the most frequent adjacent character pairs until a target vocabulary size is reached; WordPiece, used by BERT, maximises the likelihood of the training corpus given the vocabulary; and SentencePiece, used by many multilingual models, operates directly on raw Unicode so it handles any language without language-specific preprocessing.

Think of it like a publisher deciding how to divide a dictionary. Instead of one entry per full word (too many rare entries) or one entry per letter (too many steps to spell anything), the publisher finds a middle ground: common words get their own entry, rare ones are spelled out in syllable-sized chunks.

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
output = tokenizer("Tokenization affects everything downstream.")
print(output["input_ids"])
# [101, 19204, 3989, 4473, 2673, 8618, 2102, 1012, 102]
print(tokenizer.convert_ids_to_tokens(output["input_ids"]))
# ['[CLS]', 'token', '##ization', 'affects', 'everything', 'downstream', '.', '[SEP]']

The word "tokenization" is split into "token" + "##ization"; each piece consumes one slot in the Context Window.

Tokenization decisions have cascading effects: vocabulary size influences Embedding matrix size and therefore Parameter count; token fertility (how many tokens a given language requires per character) affects how efficiently multilingual models handle non-Latin scripts; and the tokenizer boundary between training and Inference must match exactly or the model sees gibberish. Quantization and model compression do not change the tokenizer, so a quantised model still uses the same vocabulary as its full-precision parent. Understanding tokenization is also essential for accurate cost estimation when using LLM APIs billed by token count.

You will encounter tokenization when debugging unexpected model behaviour on special characters, estimating prompt costs, preparing multilingual datasets, or building custom Fine-tuning pipelines that require byte-level consistency.