← AI Terminology
Momentum
Momentum is an optimisation technique that accelerates gradient descent by accumulating a velocity vector in directions of persistent gradient, dampening oscillations in directions of high curvature — enabling faster convergence than vanilla SGD.
It is a component of nearly every modern optimiser (Adam, AdamW, SGD with momentum).
It is a component of nearly every modern optimiser (Adam, AdamW, SGD with momentum).
Why It Matters in AI
Vanilla SGD takes slow, noisy steps — oscillating in ravines and creeping along flat directions. Momentum accumulates a running average of gradients, letting the optimiser build speed on consistent gradients and smooth over noise. The result is faster convergence, better navigation of flat regions, and reduced oscillation in steep directions. Without momentum, training large neural networks with SGD would be impractically slow; Adam's first-moment estimate is simply momentum with adaptive learning rates.
Key Points
| Aspect | Description |
|---|---|
| Adam | Adam's first moment (m_t) is momentum + bias correction — adaptive momentum per parameter |
| Formula | v_t = β × v_{t-1} + (1−β) × ∇L; θ = θ − α × v_t — velocity accumulates gradient history |
| Nesterov | Nesterov Accelerated Gradient (NAG): compute gradient at θ − β×v (look-ahead position) — often better |
| β (beta) | Momentum coefficient — typically 0.9 (keep 90% of previous velocity) |
| Convergence | Momentum can overshoot minima — β and LR must be tuned together |
| SGD + momentum | Still preferred over Adam for vision tasks (ResNet, ViT) — generalises better in some settings |
Simple Analogy
A ball rolling down a hill: instead of stopping at every flat spot and recalculating direction (vanilla SGD), the ball carries its speed forward (momentum). It builds up speed on consistent slopes, absorbs small bumps without stopping, and eventually rolls into the valley — faster and more smoothly than a ball that stops at every irregularity.
Common Usage Examples
torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)— standard SGD with momentumtorch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9, nesterov=True)— Nesterov variant- Adam:
torch.optim.Adam(model.parameters(), betas=(0.9, 0.999))— β₁=0.9 is the momentum coefficient - ImageNet training: SGD with momentum=0.9, cosine LR decay — still the standard for ResNet/ViT
keras.optimizers.SGD(learning_rate=0.01, momentum=0.9, nesterov=True)— Keras equivalent
Summary
In short: Momentum accelerates gradient descent by accumulating velocity in consistent gradient directions — reducing oscillation and enabling faster convergence, and is a component of Adam and every other modern optimiser.