← AI Terminology
SGD - Stochastic Gradient Descent
Stochastic Gradient Descent (SGD) is the foundational optimisation algorithm for neural networks — updating model weights using gradients computed on a small random mini-batch of data rather than the full dataset, making each update fast while providing a noisy but unbiased estimate of the true gradient.
"Stochastic" refers to the randomness of using a mini-batch instead of all data.
"Stochastic" refers to the randomness of using a mini-batch instead of all data.
Why It Matters in AI
Computing the gradient over the full dataset (batch gradient descent) is too slow for modern neural networks — one update would require a full pass over ImageNet (1.2M images). SGD approximates the gradient using 32–512 randomly selected samples per update — fast, scalable, and with noise that helps escape saddle points. With momentum and weight decay, SGD remains competitive with Adam for vision tasks (ResNet, ViT) and generalises better in some settings. All modern optimisers (Adam, AdamW) are adaptive extensions of SGD.
Key Points
| Aspect | Description |
|---|---|
| vs Adam | Adam: adaptive per-parameter LR, faster to converge; SGD+momentum: often better final generalisation |
| Nesterov | Look-ahead gradient: compute gradient at θ − β×v position — generally better than standard momentum |
| Update rule | θ ← θ − α × ∇L(θ; mini-batch) — subtract gradient estimate scaled by learning rate |
| Learning rate | Critically important for SGD — too high diverges; too low stalls. Warmup + cosine decay standard |
| With momentum | torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9) — standard configuration |
| Noise benefits | Mini-batch noise acts as regularisation — helps escape sharp minima, improves generalisation |
Simple Analogy
A hiker descending a foggy mountain by taking small, random samples of the slope under their feet (mini-batch) rather than mapping the entire mountain before moving (full batch): each step is approximate, but they reach the valley quickly and the random variation in step direction prevents them from getting stuck in small pits.
Common Usage Examples
torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=1e-4)— ImageNet trainingtorch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9, nesterov=True)— Nesterov variant- Learning rate schedule:
scheduler = CosineAnnealingLR(optimizer, T_max=90)— standard vision training - Batch size scaling: double batch → double LR — "linear scaling rule" (He et al., 2018)
keras.optimizers.SGD(learning_rate=0.01, momentum=0.9, nesterov=True)— Keras SGD
Summary
In short: SGD updates model weights using gradients from random mini-batches — the foundational optimisation algorithm that makes deep learning tractable at scale, and whose adaptive variants (Adam, AdamW) remain the standard for all modern training.