← AI Terminology
Optimizer
An optimizer is the algorithm that updates model weights during training — computing how to adjust each parameter based on gradients of the loss function to minimise the training objective, with different optimizers implementing different update rules and adaptive strategies.
The choice of optimizer significantly affects training speed, stability, and final model quality.
The choice of optimizer significantly affects training speed, stability, and final model quality.
Why It Matters in AI
Gradient descent alone — naive step in the direction of the negative gradient — is too slow and unstable for deep networks. Modern optimizers (Adam, AdamW, Lion, SOAP) add momentum, adaptive learning rates per parameter, and weight decay to navigate the complex, high-dimensional loss landscapes of neural networks efficiently. The optimizer is the engine of training: the same model and data with a different optimizer can converge 10× faster or fail entirely.
Key Points
| Aspect | Description |
|---|---|
| SGD | Stochastic Gradient Descent — simple, but requires careful LR tuning; often best for vision |
| Adam | Adaptive Moment Estimation — per-parameter adaptive LR using first and second moments |
| Lion | Symbolic optimiser (Google, 2023): uses sign of gradient update — memory efficient, often faster |
| AdamW | Adam with decoupled weight decay — fixes L2 regularization bug in Adam; standard for LLMs |
| Hyperparams | Learning rate, β₁, β₂ (momentum), ε (numerical stability), weight decay — Adam's key settings |
| Shampoo / SOAP | Second-order optimisers — use curvature information; faster convergence per step at higher cost |
Simple Analogy
A hiker with different navigation strategies: SGD follows the steepest downhill regardless of history; Adam checks a compass (momentum) and adjusts step size based on recent terrain difficulty (adaptive LR). AdamW adds a rope that pulls the hiker back toward the centre (weight decay). Different terrains (problems) favour different navigators.
Common Usage Examples
torch.optim.AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)— LLM fine-tuning standardtorch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)— ImageNet ResNet trainingtorch.optim.Adam(params, lr=3e-4, betas=(0.9, 0.999), eps=1e-8)— general Adam defaults- Lion:
pip install lion-pytorch; Lion(model.parameters(), lr=1e-4, weight_decay=1e-2) - HuggingFace:
TrainingArguments(optim="adamw_torch_fused")— fused AdamW for faster training
Summary
In short: The optimizer determines how model weights are updated during training — modern adaptive optimizers like AdamW navigate complex loss landscapes far more efficiently than vanilla gradient descent, directly determining training speed and model quality.