← AI Terminology
Warmup
Learning-rate warmup starts training with a small learning rate and ramps it up (usually linearly) over the first steps before the main schedule.
It stabilises early optimisation of deep nets and transformers.
It stabilises early optimisation of deep nets and transformers.
Why It Matters in AI
Early gradients can be large and noisy; a full LR from step 0 destabilises large models. Warmup is a standard part of transformer schedules (then cosine/linear decay).
Key Points
| Aspect | Description |
|---|---|
| Why | Avoid explosive updates while stats adapt |
| Then | Cosine decay, linear decay, constant, etc. |
| Shape | Linear warmup from 0 (or tiny) to peak LR |
| Length | Hundreds to thousands of steps; % of training |
| Related | Cosine decay, AdamW |
| Used in | BERT, GPT, LLaMA recipes universally |
Simple Analogy
Easing onto a highway instead of flooring the accelerator from a standstill — reach cruising speed gradually.
Common Usage Examples
- HF:
warmup_ratio=0.03 - LLM pretrain: thousands of warmup steps
- Plot LR schedule before long runs
- Too short warmup → early loss spikes
Summary
In short: Warmup ramps the learning rate at the start of training — a simple stabiliser used in almost every modern deep-learning schedule.