← AI Terminology
Pre-Norm vs Post-Norm
Pre-norm applies normalisation before attention/MLP; post-norm applies it after the residual add — two placements for LayerNorm/RMSNorm in a transformer block.
Modern deep LLMs overwhelmingly use pre-norm for stable training.
Modern deep LLMs overwhelmingly use pre-norm for stable training.
Why It Matters in AI
Block layout determines gradient flow through residual streams. Post-norm (original transformer) can be unstable at depth; pre-norm trains more reliably for deep stacks, which is why LLaMA-style models adopt it.
Key Points
| Aspect | Description |
|---|---|
| Extra | Many models add a final norm before the LM head |
| Quality | At scale both can work; pre-norm preferred operationally |
| Used in | LLaMA/Mistral: pre-norm + RMSNorm |
| Pre-norm | Norm then sublayer then residual add — standard in LLMs |
| Post-norm | Sublayer then Add&Norm — original Vaswani transformer |
| Stability | Pre-norm easier to train deep nets |
Simple Analogy
Post-norm seasons the whole plate after mixing; pre-norm seasons each ingredient before mixing into the residual bowl — different order, different stability when stacking many layers.
Common Usage Examples
- LLaMA: RMSNorm → Attn → +res → RMSNorm → MLP → +res
- Original Transformer: Attn/FFN then Add&Norm
- DeepNet/NormFormer on deep post-norm fixes
- Inspect where
input_layernormsits in code
Summary
In short: Pre-norm normalises before sublayers and post-norm after — modern deep LLMs use pre-norm because it trains more stably at scale.