← AI Terminology
Exploding Gradient
Exploding gradients occur during backpropagation when gradient magnitudes grow exponentially as they propagate through many layers — resulting in extremely large weight updates that destabilise training (loss diverges to NaN or oscillates wildly).
It is the complement of the vanishing gradient problem, both caused by multiplicative gradient flow through deep networks.
It is the complement of the vanishing gradient problem, both caused by multiplicative gradient flow through deep networks.
Why It Matters in AI
Exploding gradients were a major obstacle to training deep networks and RNNs before gradient clipping became standard. A single exploding gradient step can completely destroy weeks of training — weights are updated by millions of times the appropriate amount. Modern training pipelines almost universally apply gradient clipping as standard practice for this reason.
Key Points
| Aspect | Description |
|---|---|
| Cause | Chain rule: if each layer multiplies gradients by > 1, gradients grow exponentially with depth |
| Symptom | Loss spikes or NaN; gradients suddenly have very large norms in TensorBoard |
| Weight init | Poor initialisation (too large weights) triggers explosion from the first step |
| Vs vanishing | Vanishing: gradients → 0 (deep layers don't learn). Exploding: gradients → ∞ (training diverges) |
| Gradient clipping | Rescale gradient when norm > threshold: ∇ ← ∇ × (threshold / ‖∇‖) — standard fix |
| RNN vulnerability | RNNs unrolled through time have very deep effective depth — especially prone to exploding |
Simple Analogy
Compound interest applied catastrophically: if each bank in a chain of transfers charges 50% instead of subtracting 50%, the amount grows explosively. Gradients face the same problem — multiply by numbers > 1 at each layer and the final gradient is astronomical. Gradient clipping is a hard cap on how large the chain can grow.
Common Usage Examples
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)— standard after every backward passoptimizer.step()always called afterclip_grad_norm_()— prevents exploding update- TensorBoard gradient norm monitoring:
writer.add_scalar('grad_norm', total_norm, step) - LLM training: gradient clipping to 1.0 is nearly universal in HuggingFace training configs
- Transformer training:
--max_grad_norm 1.0in HuggingFace Trainer arguments
Summary
In short: Exploding gradients send weight updates to infinity — the standard fix is gradient clipping, which caps the gradient norm before each optimiser step.