The learning rate is a scalar Hyperparameter that controls the step size used to update a model's weights during each iteration of Gradient Descent. After the Loss Function is computed and backpropagation calculates the gradient of each weight, the learning rate multiplies those gradients to determine how far each weight moves in the direction that reduces loss. Too large a value causes training to overshoot minima and diverge; too small a value makes training prohibitively slow or traps the model in a poor local minimum.

Think of it like adjusting the sensitivity of a steering wheel. With too much sensitivity, every small correction sends the car fishtailing across lanes; with too little, the car barely responds and drifts off the road before you can correct it. Finding the right feel requires trial, monitoring, and often automatic adjustment mid-journey.

Modern training does not use a fixed learning rate throughout. Instead, learning rate schedules modify it over time. A common pattern is linear warmup followed by cosine decay: the rate rises from near zero over the first few thousand steps to stabilise early gradient estimates, then smoothly decreases toward the end of training:

scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=total_steps, eta_min=1e-6
)
# peak learning rate is set in the optimizer, e.g. lr=3e-4

Learning rate interacts tightly with batch size: doubling the batch size while also doubling the learning rate (the linear scaling rule) often preserves training dynamics. It is also the most commonly tuned Hyperparameter during Fine-tuning, where a much smaller value — typically 1e-5 to 1e-4 — is used to avoid overwriting a Foundation Model's pre-trained knowledge. Adaptive optimisers such as AdamW maintain per-parameter effective learning rates, reducing sensitivity to the global setting and helping prevent Overfitting.

You will encounter learning rate decisions whenever you train or fine-tune any model, and misconfigured schedules are among the most frequent causes of unstable or stalled training runs.