← AI Terminology
Xavier Initialization
Xavier initialisation (also called Glorot initialisation) sets neural network weights by sampling from a distribution scaled to keep the variance of activations constant across layers — preventing signal from vanishing or exploding at the start of training.
Introduced by Glorot & Bengio (2010); the default initialiser for layers with sigmoid/tanh activations.
Introduced by Glorot & Bengio (2010); the default initialiser for layers with sigmoid/tanh activations.
Why It Matters in AI
Randomly initialised weights that are too large cause activations to saturate and gradients to vanish; too small and the signal dies before reaching later layers. Xavier initialisation sets the scale analytically: the variance of weights is 2/(fan_in + fan_out), chosen so that the variance of activations is preserved forward and the variance of gradients is preserved backward. This makes training stable from the very first step, especially for sigmoid and tanh networks. He initialisation extends the same idea for ReLU activations.
Key Points
| Aspect | Description |
|---|---|
| Formula | Var(W) = 2 / (fan_in + fan_out) — balanced for both forward and backward signal flow |
| Use case | Sigmoid, tanh, linear activations — assumes linear activations in derivation |
| Normal variant | W ~ N(0, √(2/(fan_in+fan_out))) — Xavier normal |
| Uniform variant | W ~ Uniform(−√(6/(fan_in+fan_out)), +√(6/(fan_in+fan_out))) — Xavier uniform |
| fan_in / fan_out | fan_in = inputs to layer; fan_out = outputs from layer |
| He initialisation | Var(W) = 2/fan_in — better for ReLU, which halves the signal; PyTorch default for Conv/Linear |
Simple Analogy
Tuning a microphone's gain before a performance: too loud and the speaker distorts (exploding gradients); too quiet and the audience hears nothing (vanishing gradients). Xavier sets the gain to the mathematically correct level for the room size — signal passes cleanly through every layer of the venue without distortion or fade.
Common Usage Examples
nn.Linear(in, out)— PyTorch applies Kaiming (He) init by default; override with:torch.nn.init.xavier_uniform_(layer.weight)— Xavier uniform in-placetorch.nn.init.xavier_normal_(layer.weight)— Xavier normal in-place- Keras:
kernel_initializer="glorot_uniform"— default for Dense layers (Xavier uniform) - He init:
torch.nn.init.kaiming_uniform_(layer.weight, nonlinearity="relu")— for ReLU layers
Summary
In short: Xavier initialisation scales weight values to preserve signal variance across layers at initialisation, preventing vanishing/exploding gradients from the very first forward pass and making it the standard starting point for networks with sigmoid or tanh activations.