← AI Terminology
Temporal Difference Learning
Temporal Difference (TD) learning is a class of reinforcement learning methods that update value estimates based on the difference between successive predictions — bootstrapping from current estimates rather than waiting for the episode to end, combining ideas from Monte Carlo methods and dynamic programming.
It is the foundation of Q-learning, SARSA, and modern deep RL.
It is the foundation of Q-learning, SARSA, and modern deep RL.
Why It Matters in AI
Monte Carlo RL waits until an episode ends to update value estimates — impractical for long episodes or continuous tasks. Dynamic programming requires a model of the environment — unavailable in model-free settings. TD learning combines their advantages: update value estimates after every step using the current estimate as a target (bootstrapping), making online, incremental learning possible. Every practical deep RL algorithm — DQN, A3C, PPO, TD3 — uses TD updates at its core.
Key Points
| Aspect | Description |
|---|---|
| TD(0) | Simplest: V(sₜ) ← V(sₜ) + α[rₜ + γV(sₜ₊₁) − V(sₜ)] — update using one-step lookahead |
| TD(λ) | Eligibility traces: blend between TD(0) (λ=0, one-step) and Monte Carlo (λ=1, full return) |
| TD error | δ = rₜ + γV(sₜ₊₁) − V(sₜ) — the "surprise" signal; how wrong was the prior estimate |
| Q-learning | TD update applied to Q(s,a) — off-policy TD for action-value learning |
| Actor-Critic | Actor uses TD error as advantage signal — standard in PPO, A3C, SAC |
| Bootstrapping | Use current estimate V(sₜ₊₁) as target — no need to complete the episode before updating |
Simple Analogy
A GPS updating your ETA as you drive: instead of waiting until you arrive to know if the route was good (Monte Carlo), it updates the estimate after every turn based on current traffic (bootstrapping). Each new observation (road condition) immediately refines the prediction without requiring the full journey to complete.
Common Usage Examples
- TD(0):
V[s] += alpha * (reward + gamma * V[s_next] - V[s])— simplest value update - Q-learning:
Q[s][a] += alpha * (r + gamma * max(Q[s_next]) - Q[s][a])— TD applied to Q-values - n-step TD: balance between TD(0) and MC —
G_n = r₁ + γr₂ + ... + γⁿV(sₙ)— use n-step return stable_baselines3.A2C— advantage actor-critic using one-step TD advantage estimates- Atari DQN: replays transitions, computes TD target with target network, updates Q-network
Summary
In short: Temporal Difference learning updates value estimates after every step using bootstrapped predictions rather than waiting for episode completion — the core update mechanism underlying all practical deep RL algorithms, from Q-learning to PPO.