← AI Terminology
Experience Replay
Experience replay is a technique in deep reinforcement learning where past interactions (state, action, reward, next state) are stored in a replay buffer and randomly sampled to form training mini-batches — breaking the temporal correlation of sequential experience.
It was a key component of DQN and is used in most off-policy RL algorithms.
It was a key component of DQN and is used in most off-policy RL algorithms.
Why It Matters in AI
RL agents learn from a stream of correlated sequential experiences — bad for gradient descent, which assumes i.i.d. samples. Replay breaks this correlation: store millions of transitions, sample random mini-batches, train on them — just like supervised learning. It also makes data efficient: each transition can be used for multiple updates. Without replay, deep RL training is unstable and data-inefficient.
Key Points
| Aspect | Description |
|---|---|
| Buffer | Circular buffer of fixed size (e.g. 1M transitions) — oldest transitions discarded when full |
| Data efficiency | Each transition reused ~O(buffer/batch) times — far more efficient than online (on-policy) updates |
| Off-policy only | Requires off-policy learning (Q-learning, SAC) — on-policy methods (PPO) cannot use old transitions |
| Random sampling | Breaks temporal correlations — ensures each batch is a diverse, uncorrelated sample |
| Hindsight replay | HER: relabel failed trajectories with achieved goals — learns from failure |
| Prioritised replay | Sample transitions proportional to their TD error — focus on surprising/informative experiences |
Simple Analogy
Instead of only studying the last thing that happened in class (sequential learning), you keep a notebook of all your past exercises and randomly revise different ones each study session. Each exercise is revisited many times, and you don't over-focus on recent material. The notebook is the replay buffer.
Common Usage Examples
- DQN:
ReplayMemory(capacity=1_000_000)— deque-based buffer, sample random mini-batches of 32 stable_baselines3.SAC— uses replay buffer of 1M by default- Prioritised Experience Replay (PER):
SumTreedata structure for O(log n) priority sampling - HER (Hindsight Experience Replay):
HerReplayBufferin SB3 — relabels failed robot arm trajectories - Offline RL: pre-collected dataset treated as a fixed, non-updating replay buffer
Summary
In short: Experience replay stores past interactions in a buffer and samples them randomly for training — breaking temporal correlations and making deep RL stable and data-efficient.