← AI Terminology
Learning Rate
The learning rate is a hyperparameter that controls how large each gradient descent step is — determining how much model weights are updated after each backward pass.
It is the most important hyperparameter in neural network training.
It is the most important hyperparameter in neural network training.
Why It Matters in AI
Too high a learning rate and training diverges (loss explodes); too low and training is uselessly slow or gets stuck in suboptimal minima. The optimal learning rate sits in a narrow range and typically varies over training — warmup periods, decay schedules, and adaptive methods (Adam) all exist to manage this. Finding a good learning rate is the first step in any training run and the most common source of failed experiments.
Key Points
| Aspect | Description |
|---|---|
| Warmup | Start with a very small LR, linearly ramp up over 1–10% of training steps — prevents early instability |
| LR finder | Cyclical range test: sweep LR geometrically while training; pick the steepest loss-descent region |
| Typical range | 1e-5 to 1e-1 depending on model, batch size, and optimiser — LLM fine-tuning: 1e-5 to 2e-4 |
| Decay schedule | Cosine, linear, or step decay after warmup — reduces LR as training converges |
| Batch size link | Linear scaling rule: double batch size → double LR (for SGD); approximation, not strict |
| Adaptive methods | Adam, AdamW, RMSprop — maintain per-parameter effective learning rates, reducing sensitivity |
Simple Analogy
Adjusting the size of each step when navigating down a hill in fog: too-large steps and you overshoot the valley or tumble off a cliff; too-small steps and it takes forever. The learning rate finder is like taking test steps at different sizes to find the sweet spot before committing to the descent.
Common Usage Examples
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5)— LLM fine-tuning typical LR- LR finder:
from torch_lr_finder import LRFinder; finder.range_test(loader, end_lr=1, num_iter=100) get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=500, num_training_steps=10000)- Cyclical LR:
torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=1e-3, total_steps=1000) - Hugging Face
TrainingArguments(learning_rate=2e-5, warmup_ratio=0.1, lr_scheduler_type="cosine")
Summary
In short: The learning rate controls the step size of gradient descent — the single most critical hyperparameter, where getting it wrong means training either diverges or never converges.