← AI Terminology
Learning Rate Scheduler
A learning rate scheduler is a component that automatically adjusts the learning rate during training according to a predefined rule — typically warming up from a small value, maintaining a peak, then decaying — to improve convergence and final model quality.
Fixed learning rates are rarely optimal; schedules adapt the step size to the training phase.
Fixed learning rates are rarely optimal; schedules adapt the step size to the training phase.
Why It Matters in AI
Early training benefits from a small LR (stability during initialisation), mid-training from a large LR (fast progress), and late training from a small LR (fine-grained convergence). Cosine decay is so effective it is now the default for LLM training — a flat LR with cosine decline to near-zero. Without scheduling, models either diverge early or plateau before reaching their best achievable loss.
Key Points
| Aspect | Description |
|---|---|
| Warmup | Linear ramp from ~0 to peak LR over first N steps — prevents instability with large initial gradients |
| OneCycleLR | Warmup → peak → rapid cooldown in one cycle — fast training, effective for shorter runs |
| Step decay | Drop LR by a factor (e.g. ÷10) at fixed milestones — used in ResNet/ImageNet training |
| Cosine decay | LR follows a half-cosine curve from peak to near-zero — smooth convergence, widely used in LLMs |
| Constant + linear | Fine-tuning LLMs: constant warmup then linear decay — simpler than cosine, often sufficient |
| ReduceLROnPlateau | Reduce LR when validation metric stops improving — adaptive, requires metric to monitor |
Simple Analogy
Cruise control for learning: start slow in a school zone (warmup), cruise at highway speed (peak LR), then slow down as you approach your destination (decay). The scheduler handles the throttle automatically so the driver focuses on the road.
Common Usage Examples
scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=1000, num_training_steps=50000)torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)— cosine decay over 100 epochstorch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, patience=5, factor=0.5)— validation-aware decay- HuggingFace
TrainingArguments(lr_scheduler_type="cosine", warmup_ratio=0.05)— LLM fine-tuning torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=1e-3, steps_per_epoch=100, epochs=10)
Summary
In short: A learning rate scheduler automatically adjusts the step size during training — typically warming up for stability, then decaying for convergence — and is essential for achieving optimal model quality.