← AI Terminology
AdamW
AdamW is the Adam optimiser with decoupled weight decay — L2 regularisation applied directly to weights rather than mixed into the adaptive gradient update incorrectly.
It is the default optimiser for modern deep learning and LLM training.
It is the default optimiser for modern deep learning and LLM training.
Why It Matters in AI
Standard Adam + L2 was theoretically mismatched; AdamW fixes weight decay and improves generalisation. Nearly every LLM and vision transformer recipe lists AdamW with betas and weight decay hyperparameters.
Key Points
| Aspect | Description |
|---|---|
| Use | Transformers, diffusion, most large-scale training |
| Impl | torch.optim.AdamW |
| Origin | Loshchilov & Hutter (2017/2019) |
| Related | Adam, Lion, Adafactor, Sophia |
| Vs Adam | Decoupled weight decay vs L2-in-loss coupling |
| Defaults | β1≈0.9, β2≈0.95–0.999, wd≈0.01–0.1 in LLMs |
Simple Analogy
Adam steers with adaptive step sizes; AdamW also regularly prunes weight magnitude on a clean separate schedule — better regularisation hygiene.
Common Usage Examples
optim.AdamW(params, lr=1e-3, weight_decay=0.01)- LLM pretrain configs universally AdamW
- Tune wd separately from lr
- Compare to SGD+momentum on small CNNs
Summary
In short: AdamW is Adam with proper decoupled weight decay — the default optimiser behind modern transformer training.