← AI Terminology
Weight Decay
Weight decay is a regularisation technique that adds a penalty proportional to the squared magnitude of weights to the loss function — shrinking weights toward zero at each update step to prevent overfitting and improve generalisation.
It is equivalent to L2 regularisation and is a standard component of AdamW optimisation.
It is equivalent to L2 regularisation and is a standard component of AdamW optimisation.
Why It Matters in AI
Overfit models have large, highly tuned weights that memorise training examples instead of learning generalisable patterns. Weight decay fights this by penalising large weights — each update step slightly shrinks every weight regardless of the gradient, biasing the model toward simpler solutions. In transformers, weight decay is applied selectively (to weight matrices but not biases or LayerNorm params). AdamW — the dominant LLM optimiser — decouples weight decay from the adaptive learning rate, fixing a subtle bug in the original Adam implementation.
Key Points
| Aspect | Description |
|---|---|
| Loss term | L_total = L_task + λ · ‖W‖² — λ (weight decay coefficient) controls regularisation strength |
| Adam vs AdamW | Adam absorbs weight decay into the adaptive denominator (wrong); AdamW applies it directly |
| What to decay | Weight matrices only; skip biases, LayerNorm γ/β, embedding tables — standard practice |
| L2 equivalence | Weight decay = L2 regularisation for SGD; they differ for adaptive optimisers like Adam |
| Typical values | λ = 0.01–0.1 for LLMs; 1e-4 for CNNs — tuned as a hyperparameter |
| Parameter update | w ← w · (1 − lr·λ) − lr·∂L/∂w — weight shrinks by factor (1 − lr·λ) each step |
Simple Analogy
A rubber band attached to each weight, pulling it back toward zero after every gradient step. Useful weights overcome the pull (their gradients are strong enough); useless weights that are large simply from memorisation get snapped back to small values. The rubber band's stiffness is λ — tighter means more aggressive regularisation.
Common Usage Examples
torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.01)— standard LLM optimisertorch.optim.SGD(model.parameters(), lr=0.1, weight_decay=1e-4, momentum=0.9)— CNN training- HuggingFace:
TrainingArguments(weight_decay=0.01)— applied automatically in Trainer - Selective decay:
optimizer_grouped_parameters = [{"params": decay_params, "weight_decay": wd}, {"params": no_decay_params, "weight_decay": 0.0}] - Check:
print(optimizer.param_groups[0]["weight_decay"])— verify weight decay is set
Summary
In short: Weight decay penalises large weights by continuously shrinking them toward zero — a simple but powerful regulariser that prevents memorisation and is a standard component of AdamW, the dominant optimiser for training modern language models.