← AI Terminology
Gradient Descent
Gradient descent is the optimisation algorithm that trains neural networks by iteratively moving model parameters in the direction that most reduces the loss — computed as the negative gradient of the loss function with respect to the parameters.
It is the foundational algorithm underlying all neural network training.
It is the foundational algorithm underlying all neural network training.
Why It Matters in AI
Every neural network trained today uses gradient descent or a variant of it. The gradient tells the model which direction to move each weight to reduce the error; multiplying by the learning rate controls the step size. Understanding gradient descent explains why learning rates matter, why local minima are less of a problem in high dimensions than feared, and why modern training requires careful scheduling and normalisation.
Key Points
| Aspect | Description |
|---|---|
| Batch GD | Gradient computed over full dataset — stable but slow; rarely used in practice |
| Landscape | High-dim loss surfaces have many saddle points; gradient descent escapes them via noise |
| Extensions | Momentum, Adam, AdaGrad — all add memory or adaptivity on top of basic gradient descent |
| Mini-batch | Gradient from a mini-batch (32–512) — standard; balances speed and stability |
| Update rule | θ ← θ − α∇_θ L(θ) — move parameters opposite to the gradient by learning rate α |
| Stochastic (SGD) | Gradient from one example — fast but very noisy |
Simple Analogy
Descending a foggy mountain blindfolded: at each step, you feel which direction is steepest downhill (gradient), then take a small step in that direction (learning rate × gradient). You might zigzag, might reach a valley rather than the lowest point, but you consistently move downward. Larger steps are risky (overshoot); smaller steps are safe but slow.
Common Usage Examples
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)- Training loop:
loss.backward(); optimizer.step(); optimizer.zero_grad() - Gradient descent on a quadratic: exact minimum in one step if LR = 1/(2L); oscillates if LR too high
- Learning rate finder:
lr-finderpackage sweeps LR and plots loss — find steepest descent region - Cosine decay:
lr = lr_0 * 0.5 * (1 + cos(π * t / T))— standard schedule for LLM fine-tuning
Summary
In short: Gradient descent trains neural networks by moving weights downhill on the loss landscape — the simple iterative procedure that, repeated billions of times, produces intelligent systems.