Gradient descent is the iterative optimization algorithm used to train nearly every modern neural network. At each step, it computes the gradient of the loss function with respect to all model parameters — a vector pointing in the direction of steepest increase in error — and then nudges the parameters a small step in the opposite direction. The size of that step is governed by the learning rate hyperparameter. Repeating this process over many batches of training data gradually drives the model toward a configuration that minimizes loss. Variants include stochastic gradient descent (SGD, one random example per step), mini-batch gradient descent (the standard in practice), and adaptive-rate optimizers like Adam, AdaFactor, and Lion that scale the step size per parameter.

Think of it like finding the lowest point in a dark, foggy mountain range by feel. At each step you test the slope of the ground beneath your feet, then take a small step downhill. You cannot see the full landscape, but by consistently moving in the locally downward direction you eventually reach a valley — though not necessarily the absolute lowest point in the entire range.

In PyTorch, a single training step involves computing the gradient and applying the optimizer:

optimizer.zero_grad()       # clear stale gradients
loss = criterion(model(X), y)  # forward pass
loss.backward()             # compute gradients via autograd
optimizer.step()            # update parameters: w -= lr * grad

Gradient descent is inseparable from Loss Function (what it minimizes), Backpropagation (how gradients are computed efficiently), Hyperparameter (learning rate is the most critical one), and Overfitting (descending too far on training data). RLHF replaces the supervised loss with a reward signal but still uses gradient descent under the hood.

You will encounter gradient descent in every neural network training tutorial, in debugging sessions where loss curves plateau or diverge, and in discussions of optimizer choice for Fine-tuning large pretrained models.