Softmax is a mathematical function that takes a vector of arbitrary real numbers and transforms it into a valid probability distribution: every output value falls between 0 and 1, and all values sum to exactly 1. The function amplifies the largest input relative to the others—an effect controlled by a temperature parameter—making it the standard final layer in classification networks and the mechanism by which language models express confidence over a vocabulary at each generation step. Mathematically, each element is computed as the exponential of that input divided by the sum of exponentials across all inputs.

Think of it like converting a panel of judges' raw scores into weighted votes. If three judges award 8, 2, and 1 points, softmax doesn't just divide by 11—it first exponentiates each score, dramatically boosting the leader, then normalizes, so the top judge ends up wielding far more than a proportional share of the final decision.

In a language model generating text, the last linear layer produces a vector of logits, one per vocabulary token. Applying softmax converts those logits to probabilities: probs = torch.softmax(logits / temperature, dim=-1). Setting Temperature below 1.0 sharpens the distribution toward the top token; values above 1.0 flatten it, increasing diversity.

Softmax is deeply linked to Attention—the attention score matrix is also normalized with softmax—and to Activation Function theory; understanding it is prerequisite to grasping Self-Attention, Loss Function design, and Temperature sampling. You will encounter it in virtually every classification model, every language model decoder, and every tutorial on neural network outputs.