← AI Terminology

Batch Normalization

Batch Normalization is a technique that normalises a layer's inputs to zero mean and unit variance across the current mini-batch, then applies learned scale and shift parameters.

Introduced by Ioffe & Szegedy (2015), it dramatically stabilises and accelerates training of deep networks.
Why It Matters in AI
Deep networks suffer from "internal covariate shift" — the distribution of each layer's inputs changes as earlier weights update, forcing each layer to constantly readjust. BatchNorm fixes this by normalising activations at each layer, allowing much higher learning rates and making training far less sensitive to weight initialisation. It also acts as a mild regulariser, often reducing the need for dropout.
Key Points
Aspect Description
Variants Layer Norm (LLMs), Instance Norm (style transfer), Group Norm (detection with small batches)
Limitation Fails at batch size = 1 or very small batches — Layer Norm or Group Norm preferred there
Normalisation μ and σ computed per feature across the batch; output = (x − μ) / σ × γ + β
Inference mode Uses running mean/variance accumulated during training, not the current batch statistics
Where to place Typically after the linear layer and before the activation function
Learnable params γ (scale) and β (shift) — let the network undo normalisation if beneficial
Simple Analogy
Every morning, a new team of workers arrives with different energy levels. BatchNorm is the manager who normalises everyone to the same starting pace before the day begins, so the assembly line never gets thrown off by one very slow or very fast worker.
Common Usage Examples
  • nn.BatchNorm2d(num_features) in PyTorch — standard in CNN architectures (ResNet, EfficientNet)
  • model.train() vs model.eval() — must call eval() to switch BatchNorm to inference statistics
  • keras.layers.BatchNormalization() placed after Dense or Conv2D layers
  • ResNet paper used BatchNorm to train 100+ layer networks that were previously untrainable
  • Replaced by Layer Normalization in Transformer-based models (GPT, BERT, Llama)
Summary
In short: Batch Normalization stabilises deep network training by normalising each layer's inputs — it's what made very deep CNNs practical to train.