← AI Terminology
RMSNorm - Root Mean Square Layer Normalization
RMSNorm normalises activations by their root-mean-square, without mean-centering: y = x / RMS(x) · g, with a learned gain g.
It is cheaper than LayerNorm and is the default normaliser in LLaMA, Mistral, Gemma, and many recent LLMs.
It is cheaper than LayerNorm and is the default normaliser in LLaMA, Mistral, Gemma, and many recent LLMs.
Why It Matters in AI
Every transformer block applies normalisation; at LLM scale, dropping mean subtraction saves compute and often matches LayerNorm quality. Open-weight model code almost always uses RMSNorm.
Key Points
| Aspect | Description |
|---|---|
| Origin | Zhang & Sennrich (2019) |
| Params | Gain vector g only (often no bias) |
| Formula | RMS(x) = sqrt(mean(x²) + ε); y = x/RMS · g |
| Used in | LLaMA family, Mistral, Qwen, DeepSeek |
| Placement | Typically pre-norm before attention/MLP |
| Vs LayerNorm | No mean centering — only scale by RMS; fewer ops |
Simple Analogy
LayerNorm re-centers and re-scales; RMSNorm only re-scales loudness to a standard level without shifting the center of the signal.
Common Usage Examples
LlamaRMSNormin Hugging Face Transformers- Fused RMSNorm CUDA kernels
- ε typically 1e-5 or 1e-6
- Replace
nn.LayerNormwhen porting modern LLMs
Summary
In short: RMSNorm is the simpler, faster LayerNorm alternative that normalises by RMS only — now the default in most modern LLMs.