Perplexity is an information-theoretic metric that quantifies how confidently a language model predicts a held-out text sequence. Formally, it is the exponentiated average negative log-likelihood per token: if the model assigns probability p to each token given its context, perplexity equals 2 raised to the power of the average cross-entropy, or equivalently exp(−(1/N) Σ log p(t_i)). A perplexity of 10 means the model is, on average, as uncertain as if it had to choose uniformly among 10 equally likely next tokens at each position. Lower perplexity indicates a model that assigns higher probability to the correct tokens — it is less "surprised" by the text.

Think of it like a weather forecaster's track record compressed into a single number. A forecaster who almost always says "70% chance of rain" when it actually rains, and "20% chance" when it is sunny, has low perplexity — their predictions are well-calibrated and rarely surprised. A forecaster who confidently predicts sunshine the day a hurricane arrives has very high perplexity.

Computing perplexity in PyTorch on a small corpus demonstrates the mechanics:

import torch, math
from transformers import GPT2LMHeadModel, GPT2TokenizerFast

model = GPT2LMHeadModel.from_pretrained("gpt2")
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
inputs = tokenizer("The cat sat on the mat.", return_tensors="pt")
with torch.no_grad():
    loss = model(**inputs, labels=inputs["input_ids"]).loss
print(f"Perplexity: {math.exp(loss.item()):.2f}")

Perplexity is useful for comparing models trained on the same domain and for detecting distribution shift — a model's perplexity spikes when it encounters text very different from its training data, which is a signal used in some RAG and Hallucination detection pipelines.

Perplexity is directly shaped by the Loss Function used during Pretraining (cross-entropy over next-token predictions), and it is meaningfully compared only across models evaluated on identical tokenizations — differences in Tokenization schemes make raw perplexity numbers incomparable across model families.

You will encounter perplexity most often in research papers benchmarking language models on standard corpora like WikiText-103 or Penn Treebank, and in production when monitoring model quality on domain-specific validation sets.