← AI Terminology
Cosine Decay
Cosine decay anneals the learning rate following a cosine curve from a peak down toward a minimum over training (often after warmup).
It is the default LR schedule for many LLM and vision trainings.
It is the default LR schedule for many LLM and vision trainings.
Why It Matters in AI
Constant LR wastes late training; abrupt drops are crude. Cosine annealing smoothly lowers LR and often improves final quality. Reading training configs means recognising warmup+cosine.
Key Points
| Aspect | Description |
|---|---|
| Alt | Linear decay, inverse sqrt, WSD schedules |
| Pair | Almost always after linear warmup |
| Formula | LR follows cosine from η_max to η_min over T steps |
| Related | Warmup, one-cycle policy |
| Used in | ViT, LLaMA-style pretraining, many finetunes |
| Variants | Cosine with restarts; half-cycle only |
Simple Analogy
Gradually dimming studio lights along a smooth curve instead of a harsh switch-off — softer landing into the final epochs.
Common Usage Examples
CosineAnnealingLR/ HF cosine schedule- Pretrain plots: warmup spike then cosine tail
- Set
lr_endmin ratio - Compare final loss vs linear decay
Summary
In short: Cosine decay smoothly lowers the learning rate along a cosine curve — the standard late-training schedule for modern nets.