Gradient Descent is the core optimisation algorithm used to train most machine learning models. It gradually adjusts the model’s parameters to reduce its errors, step by step.
What it means in plain English
Training a model means finding the parameter values that make its predictions as accurate as possible. Gradient descent does this by repeatedly nudging the parameters in the direction that most reduces the error, a little at a time. Picture walking downhill in fog: you feel which way is steepest down and take a step, repeating until you reach the bottom. The “bottom” here is the point of lowest error.
Each step is guided by the gradient — a signal for how to change each parameter to improve — and the step size is set by the learning rate.
A simple example
A model starts with random parameters and makes bad predictions. Gradient descent measures the error, works out how to tweak each parameter to reduce it, and takes a small step. After many thousands of steps, the parameters settle into values that make good predictions.
Why it matters
Gradient descent is the engine that actually does the “learning” in machine learning. Nearly every neural network, including large language models, is trained using some form of it, which is why it is one of the most fundamental ideas in the field.
Related terms
- Backpropagation — computes the gradients gradient descent uses.
- Loss Function — measures the error being minimised.
- Learning Rate — sets the size of each step.
Frequently asked questions
What is gradient descent?
It is the optimisation method that trains most models: it repeatedly adjusts the model’s parameters in the direction that reduces error, gradually finding better values.
What is the learning rate’s role?
The learning rate controls how big each adjustment step is — too large and training can overshoot or diverge, too small and it trains very slowly.
