← AI Terminology
Adam Optimizer
Adam (Adaptive Moment Estimation) is an optimisation algorithm that adapts the learning rate for each parameter individually, combining momentum and RMSprop into a single update rule.
It was introduced by Kingma & Ba in 2014 and is the default optimiser for most deep learning training runs.
It was introduced by Kingma & Ba in 2014 and is the default optimiser for most deep learning training runs.
Why It Matters in AI
Adam converges faster than vanilla SGD on most tasks because it maintains per-parameter learning rates that automatically scale up or down based on gradient history. Its robustness to hyperparameter choices made it the practical default — most practitioners start with Adam and only switch if there is a specific reason not to.
Key Points
| Aspect | Description |
|---|---|
| Core idea | Tracks first moment (mean of gradients) and second moment (uncentered variance) per parameter |
| Limitation | Higher memory use than SGD (stores two moment vectors per parameter) |
| Competitors | SGD+momentum (better generalisation on CV), Adafactor (memory-efficient for LLMs), Lion |
| Key variant | AdamW — decouples weight decay from gradient updates; preferred for Transformer training |
| Learning rate | Effective rate adapts per parameter: large for rare features, small for frequent ones |
| Hyperparameters | lr (default 1e-3), β₁=0.9, β₂=0.999, ε=1e-8 |
Simple Analogy
Adam is like a hiker who tracks not just which direction is downhill right now, but also how steep the terrain has been recently. Parameters that have seen consistent gradients get smaller steps (already moving well); parameters with rare or noisy signals get bigger corrective steps.
Common Usage Examples
torch.optim.Adam(model.parameters(), lr=1e-3)— standard PyTorch setuptorch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=0.01)— standard for Transformers- HuggingFace
Trainerdefaults to AdamW with linear warmup - BERT, GPT-2, T5 all trained with Adam/AdamW
tf.keras.optimizers.Adam()in TensorFlow/Keras
Summary
In short: Adam is the workhorse optimiser of deep learning — adaptive, fast to converge, and almost always the right starting point.