A loss function (also called a cost function or objective function) is a mathematical expression that quantifies the discrepancy between a model's predictions and the desired ground-truth outputs. During training, the optimiser's job is to adjust the model's weights so that the loss function's value decreases over successive batches of data. The shape of the loss landscape — how it rises and falls across the high-dimensional parameter space — determines how easy or difficult optimisation will be, and the choice of loss function is one of the most consequential design decisions in building any model.

Think of it like a GPS navigation score that adds penalties every time the driver takes a wrong turn or adds unnecessary distance. The driver (the optimiser) consults this score after every decision and gradually learns routes that keep the penalty as low as possible.

Different tasks require different loss functions. Cross-entropy loss is standard for classification and for next-token prediction in language models, because it directly penalises low probability assigned to the correct class. Mean-squared error (MSE) suits regression tasks where the output is a continuous value. Contrastive loss pulls embeddings of similar items together and pushes dissimilar ones apart, which is how models like CLIP learn to align images with text:

# Cross-entropy for a batch of token predictions
import torch.nn.functional as F
loss = F.cross_entropy(logits.view(-1, vocab_size), targets.view(-1))

# MSE for a regression head
loss = F.mse_loss(predictions, targets)

The loss function works in conjunction with Gradient Descent, which uses the function's derivatives to determine the direction and magnitude of each weight update. Poorly chosen loss functions contribute to Overfitting when the metric optimised during training does not align with real-world performance. Perplexity is directly derived from cross-entropy loss in language modelling tasks. RLHF introduces an additional human-preference reward signal that acts as an auxiliary loss during policy fine-tuning, supplementing the base Pretraining objective.

You will encounter loss function choices whenever you are training a model from scratch, adapting a pre-trained model with Fine-tuning, or debugging why a model is not improving despite running many epochs.