← AI Terminology
Saddle Point
A saddle point is a critical point (zero gradient) in the loss landscape of a neural network where the loss is a local minimum in some directions and a local maximum in others — resembling a horse saddle that curves up front-to-back and down side-to-side.
Saddle points are far more common than local minima in high-dimensional neural network loss landscapes.
Saddle points are far more common than local minima in high-dimensional neural network loss landscapes.
Why It Matters in AI
Early neural network theory assumed local minima were the main obstacle to convergence — a model stuck in a local minimum couldn't improve. Research (Dauphin et al., 2014; Goodfellow et al., 2015) showed the real obstacle in high dimensions is saddle points: the probability of all dimensions being in a local minimum simultaneously decreases exponentially with dimension, but saddle points (some directions ascending, some descending) are exponentially common. Stochastic gradient descent (with noise from mini-batches) naturally escapes saddle points; momentum and Adam help further.
Key Points
| Aspect | Description |
|---|---|
| Momentum | Adam and SGD+momentum escape saddle points faster by using velocity to cross flat regions |
| Definition | Gradient = 0, but Hessian has both positive and negative eigenvalues — not a min or max |
| Plateau problem | Near saddle points, gradients are very small — training slows dramatically ("flat region") |
| Escape mechanism | SGD noise from mini-batches naturally perturbs away from saddle points over time |
| High-dim frequency | In n-dimensional space, almost all critical points are saddle points; local minima are rare |
| Saddle-free Newton | Algorithm that explicitly uses Hessian curvature to escape saddle points — computationally expensive |
Simple Analogy
A mountain pass: standing at the pass, you're at the lowest point between the two peaks (minimum in one direction) but the highest point along the valley floor (maximum in another direction). Gradient is zero — you could mistake it for a valley floor. A rolling ball (SGD with momentum) naturally rolls away down the valley even when the surface appears locally flat.
Common Usage Examples
- Detecting: Hessian eigenvalue analysis at critical points — all negative = local min; mixed = saddle point
- Perturbed SGD: add small Gaussian noise to gradient when it becomes very small — escapes flat regions
- Adam: second moment estimate prevents tiny updates at saddle points — naturally escapes
- Visualisation:
loss_landscapeslibrary — plot 2D slices of neural network loss surface - Research: Dauphin et al. 2014 "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization"
Summary
In short: Saddle points are far more common than local minima in deep learning loss landscapes — points where the gradient is zero but the loss can still decrease in some directions, with mini-batch noise and momentum being the primary mechanisms for escaping them.