Gradient Descent
An optimisation algorithm that improves a model by repeatedly nudging its parameters in the direction that most reduces error, following the slope of the loss function downhill.
Gradient descent is how machine learning models learn: measure the error, calculate the slope (gradient) of the error with respect to each parameter, then take a small step in the opposite direction. Repeat until the error stops improving.
The size of each step is the learning rate, and it is one of the most consequential settings in training. Too large and the model overshoots, oscillates, or diverges; too small and training takes forever or gets stuck in a poor region.
Important variants:
- Batch gradient descent: uses the entire dataset per step — accurate but slow.
- Stochastic gradient descent (SGD): uses one example at a time — noisy but fast.
- Mini-batch SGD: the practical default, balancing stability and speed.
- Momentum and adaptive methods (Adam, AdamW): accumulate past gradients or scale steps per parameter to converge faster.
For deep networks, gradient descent is paired with backpropagation, which supplies the gradients. Modern training runs add schedulers that decay the learning rate over time, plus regularisation techniques that keep the model from memorising its training data.