← AI Terminology
Convergence
Convergence in machine learning refers to the state where a model's training loss (and validation loss) stabilise and stop improving significantly with further training — the optimisation process has found a (local or global) minimum.
A training run that hasn't converged is still improving; one that has converged has reached a stable state.
A training run that hasn't converged is still improving; one that has converged has reached a stable state.
Why It Matters in AI
Convergence tells you when training is done and whether training succeeded at all. Failure to converge (loss oscillates or diverges) signals a problem: learning rate too high, gradient explosion, bad data, or architectural issues. Over-training past convergence on training loss while validation loss rises is overfitting. Understanding convergence is essential for debugging training runs and deciding when to stop.
Key Points
| Aspect | Description |
|---|---|
| LLM training | Pre-training runs of months rarely reach full convergence — Chinchilla laws define compute-optimal stopping |
| Loss plateau | Primary signal: training loss curve flattens — further steps yield diminishing returns |
| Learning rate | Too high: loss diverges or oscillates. Too low: convergence is extremely slow |
| Local minimum | Gradient descent may converge to a local minimum — not necessarily the global best |
| Saddle points | More common than local minima in high dimensions — gradients are near zero but not a minimum |
| Early stopping | Stop when validation loss stops improving — prevents overfitting post-convergence on train set |
Simple Analogy
Hiking down a mountain in fog: convergence is when you stop descending and your feet find flat ground. You might be in a valley (local minimum) or on a wide plain (global minimum) — but you've stopped making downward progress. The learning rate is how large your steps are; too large and you overshoot; too small and you never arrive.
Common Usage Examples
- TensorBoard training curves: plot loss/step — flat curve = convergence
EarlyStopping(monitor='val_loss', patience=5)in Keras — stops when validation loss plateaus- Learning rate finder: increase LR until loss diverges, then train at 10× smaller — ensures convergence
- Warmup schedule: start with small LR, ramp up, then decay — improves convergence stability for LLMs
- Loss spikes during LLM training: common for 10–30B parameter models — resolved with gradient clipping
Summary
In short: Convergence is when training stops improving — the optimisation has settled, and the question becomes whether it settled in a good place.