← AI Terminology
Bellman Equation
The Bellman equation expresses the value of a state (or state-action pair) in reinforcement learning as the immediate reward plus the discounted value of the next state — decomposing the total future reward into a recursive, one-step relationship.
It is the mathematical foundation of Q-learning, temporal difference learning, and dynamic programming in RL.
It is the mathematical foundation of Q-learning, temporal difference learning, and dynamic programming in RL.
Why It Matters in AI
The Bellman equation converts the intractable problem of "what's the total reward from here to the end of time?" into a tractable recursive update: you only need to estimate one step ahead and bootstrap from your current value estimate. This is why TD-learning, DQN, and PPO all work — they are all solving or approximating the Bellman equation.
Key Points
| Aspect | Description |
|---|---|
| DP methods | Value iteration, policy iteration — solve Bellman equations exactly when model is known |
| Q-function | Q(s,a) = r + γ max_a' Q(s',a') — action-value form; DQN learns this directly |
| TD learning | Solves Bellman equation approximately from experience without a model of the environment |
| Bellman optimality | V*(s) = max_a[r(s,a) + γV*(s')] — optimal policy chooses action maximising this expression |
| Discount factor γ | 0 < γ < 1 — how much future rewards are worth relative to immediate reward (0=myopic, 1=far-sighted) |
| Bellman expectation | V(s) = E[r + γV(s')] — value of state s is expected reward plus discounted next-state value |
Simple Analogy
A chess player evaluates a position not by calculating every possible game to the end, but by estimating: "this position is worth +0.5 pawns, and after my move I expect to reach a position worth +0.8 — so this move is good." The Bellman equation formalises this one-step lookahead with discounted future value.
Common Usage Examples
- Q-learning update:
Q(s,a) ← Q(s,a) + α[r + γ max_a' Q(s',a') − Q(s,a)] - DQN: neural network approximates Q(s,a) — minimises TD error derived from Bellman equation
- Value iteration: repeatedly apply Bellman optimality operator until V(s) converges
- Policy gradient methods use Bellman-derived value estimates as baselines (in actor-critic)
- AlphaGo: MCTS + neural network value function approximate Bellman-optimal play in Go
Summary
In short: The Bellman equation is the recursive insight that makes reinforcement learning tractable — you don't need to see the whole future, just estimate one step at a time.