← AI Terminology
Layer Normalization
Layer normalization is a technique that normalises activations across the feature dimension of each individual training example — computing mean and variance over all neurons in a layer for that example and rescaling — stabilising training without dependence on batch size.
It is the standard normalisation in transformers, replacing batch normalization for sequence models.
It is the standard normalisation in transformers, replacing batch normalization for sequence models.
Why It Matters in AI
Transformers cannot use batch normalization because sequence lengths vary and batch statistics are unreliable for small batches or single-example inference. Layer norm operates per-example, per-layer — making it batch-size independent and ideal for language model training. Without it, deep transformers would suffer unstable gradients and slow convergence. Every major LLM (GPT, BERT, Llama, Claude) uses layer norm as a core building block.
Key Points
| Aspect | Description |
|---|---|
| Formula | y = (x − μ) / (σ + ε) × γ + β — μ, σ computed over features; γ, β are learned parameters |
| Pre-norm | Modern transformers apply layer norm before (not after) attention and FFN sub-layers (pre-LN) |
| RMS Norm | Simplified variant used in Llama/GPT-NeoX — drops the mean subtraction, 10–15% faster |
| Inference | Parameters γ, β are small and add negligible compute — no batch size dependency at inference |
| Gradient flow | Normalisation prevents activations from growing unboundedly — stabilises very deep networks |
| vs Batch Norm | Batch norm: normalise across batch dimension; Layer norm: normalise across feature dimension per example |
Simple Analogy
Grading on a curve per student: instead of adjusting scores based on how the whole class performed (batch norm), each student's scores are rescaled based only on their own performance across all subjects — making the adjustment independent of who else is in the room.
Common Usage Examples
nn.LayerNorm(d_model)in PyTorch — standard layer norm used in every transformer implementationx = self.norm(x + self.attention(x))— pre-norm transformer block pattern (post-residual connection)- Llama 3 uses RMSNorm:
class RMSNorm: return x * (1 / torch.sqrt(mean(x**2) + eps)) * weight - BERT:
BertLayerNormapplied after each attention and FFN sub-layer keras.layers.LayerNormalization(axis=-1)— Keras equivalent
Summary
In short: Layer normalization normalises each example's activations across the feature dimension, independently of batch size — the essential stability technique that makes training deep transformers practical.