← AI Terminology
He Initialization
He initialization (also called Kaiming initialization) is a weight initialisation scheme for neural networks that sets initial weights by sampling from a distribution with variance 2/n_in — specifically designed for layers followed by ReLU activations, ensuring stable gradient flow from the start of training.
Introduced by Kaiming He et al. (2015) to enable training of very deep residual networks.
Introduced by Kaiming He et al. (2015) to enable training of very deep residual networks.
Why It Matters in AI
The variance of a neural network's activations depends on how weights are initialised. Too large: activations explode; too small: they vanish — in both cases gradients become uninformative before even one step of training. He initialisation is specifically calibrated for ReLU-activated networks (which zero out half the neurons), ensuring layer outputs have unit variance at initialisation. It enabled ResNet-like deep networks to train reliably.
Key Points
| Aspect | Description |
|---|---|
| Effect | Ensures signal neither vanishes nor explodes through depth at initialisation |
| Formula | W ~ N(0, 2/n_in) — normal distribution with variance 2/n_in where n_in = fan-in of the layer |
| Why 2, not 1 | ReLU zeros out ~half the activations — factor of 2 compensates to maintain unit variance |
| PyTorch default | nn.Linear and nn.Conv2d use Kaiming Uniform by default since PyTorch 1.0 |
| Xavier (Glorot) | Alternative for sigmoid/tanh: W ~ N(0, 2/(n_in + n_out)) — not optimal for ReLU |
| For Transformers | Pre-LN Transformers sometimes use scaled Xavier; post-LN Transformers less sensitive to init |
Simple Analogy
Setting the volume knob on 100 stacked amplifiers: if every amp doubles the signal (init too large), the final output is deafening noise; if every amp halves it (init too small), you hear nothing. He initialisation sets each amp to exactly compensate for ReLU's signal reduction — the stack passes sound through cleanly from the first moment.
Common Usage Examples
- PyTorch default:
nn.Conv2dinitialised withkaiming_uniform_(a=0, mode='fan_in', nonlinearity='relu')automatically - Explicit:
torch.nn.init.kaiming_normal_(layer.weight, mode='fan_in', nonlinearity='relu') - Xavier alternative:
torch.nn.init.xavier_uniform_(layer.weight)— for Tanh/Sigmoid activations - ResNet paper used He init to enable stable training of 152-layer networks for the first time
- Custom init:
for m in model.modules(): if isinstance(m, nn.Linear): kaiming_normal_(m.weight)
Summary
In short: He initialisation sets weights to maintain stable signal variance at the start of training for ReLU networks — the default in PyTorch and the reason very deep networks can be trained without explosion or vanishing from epoch 1.