← AI Terminology
RMSprop
RMSprop (Root Mean Square Propagation) is an adaptive learning rate optimiser that divides each parameter's gradient by a running average of recent squared gradients — adapting the effective learning rate per parameter based on its recent gradient magnitude, preventing large updates for parameters with frequently large gradients.
It was proposed by Geoff Hinton and is the direct precursor to Adam.
It was proposed by Geoff Hinton and is the direct precursor to Adam.
Why It Matters in AI
Vanilla gradient descent uses the same learning rate for every parameter — too large for parameters with large gradients (divergence) and too small for parameters with small gradients (slow convergence). RMSprop solves this by tracking a moving average of squared gradients per parameter and normalising the update — effectively reducing the LR for frequently updated parameters and increasing it for rarely updated ones. This adaptive approach enables much faster convergence, particularly for non-stationary objectives like RNNs and online learning.
Key Points
| Aspect | Description |
|---|---|
| Formula | E[g²]_t = ρ × E[g²]_{t-1} + (1-ρ) × g²_t; θ ← θ − α / √(E[g²]_t + ε) × g_t |
| vs Adam | Adam = RMSprop + momentum (first moment estimate) + bias correction — typically outperforms RMSprop |
| vs Adagrad | Adagrad accumulates all past squared gradients (monotonically increasing denominator); RMSprop uses exponential moving average |
| Best use cases | RNN training, online learning, non-stationary problems — where Adam is often comparable |
| ρ (decay rate) | Typically 0.9 — exponential moving average of squared gradients |
| Original proposal | Geoff Hinton, Coursera lecture slide 29 (2012) — unpublished but widely cited and adopted |
Simple Analogy
A marathon runner adjusting pace: instead of running at constant speed (fixed LR), they slow down on recent steep hills (high gradient parameters) and speed up on recent flat terrain (low gradient parameters). RMSprop tracks which "muscles" have been working hardest recently and gives those parameters a smaller update nudge.
Common Usage Examples
torch.optim.RMSprop(model.parameters(), lr=1e-3, alpha=0.9, eps=1e-8)— standard PyTorch RMSprop- Keras:
optimizer = keras.optimizers.RMSprop(learning_rate=1e-3, rho=0.9)— Keras RMSprop - RNN training: historically preferred over Adam for LSTM language models in some settings
keras.optimizers.RMSprop(learning_rate=0.001, centered=True)— centered variant normalises by variance- AlphaGo: early training used RMSprop for policy and value network optimisation
Summary
In short: RMSprop adapts the learning rate per parameter by dividing by a moving average of recent squared gradients — the adaptive optimiser that solved Adagrad's learning rate decay problem and directly inspired Adam, which is now the standard.