← AI Terminology
Vanishing Gradient
The vanishing gradient problem occurs during backpropagation when gradients shrink exponentially as they flow backward through many layers — becoming so small that early layers receive near-zero updates and effectively stop learning.
It was the primary barrier to training deep networks before ReLU and residual connections.
It was the primary barrier to training deep networks before ReLU and residual connections.
Why It Matters in AI
In deep networks, each backward pass multiplies gradients by the derivative of each activation function. Sigmoid and tanh derivatives are bounded well below 1 — stacking 10+ layers multiplies these fractions together, producing gradients near zero at early layers. This is why networks deeper than ~5 layers failed to train effectively before 2012. Solutions — ReLU, residual connections, batch norm, LSTM gates, and careful initialisation — collectively solved this problem and enabled modern deep learning.
Key Points
| Aspect | Description |
|---|---|
| LSTM/GRU | Gated cell state carries gradient over many timesteps without multiplicative decay |
| ReLU fix | Gradient = 1 for positive inputs — no saturation; gradient flows freely |
| Root cause | Chain rule multiplication of sub-1 values over many layers → exponential decay |
| Residual fix | Skip connections create gradient highways that bypass layer-by-layer decay |
| Sigmoid/tanh | Max gradient ≤ 0.25 / 1.0 at saturation — problematic at both extremes |
| Xavier/He init | Proper weight initialisation prevents saturation at the start of training |
Simple Analogy
A game of telephone across 20 people: each person whispers to the next, losing a bit of volume each time. By person 20, the message is inaudible — the first person receives no feedback. ReLU is like giving each person a microphone: the signal doesn't decay no matter how many people pass it along.
Common Usage Examples
- Visualise:
grad_norm = [p.grad.norm().item() for p in model.parameters()]— plot per layer - ReLU replacement:
nn.Sigmoid()→nn.ReLU()— standard fix for feedforward nets - Residual block:
out = F.relu(self.conv(x)) + x— shortcut carries full gradient - LSTM:
nn.LSTM(input_size, hidden_size)— cell state preserves gradient over 100+ timesteps - Gradient clipping:
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)— controls exploding gradient (opposite problem)
Summary
In short: Vanishing gradients — where backpropagated signals decay to zero across deep layers — were the central obstacle to deep learning, solved by ReLU activations, residual connections, LSTM gates, and initialisation schemes that together enabled training networks of arbitrary depth.