← AI Terminology
Value Function
A value function in reinforcement learning estimates the expected cumulative discounted reward an agent will receive from a given state (or state-action pair) onward, under a particular policy — encoding how "good" it is to be in that state.
It is the foundation of dynamic programming, Q-learning, and actor-critic algorithms.
It is the foundation of dynamic programming, Q-learning, and actor-critic algorithms.
Why It Matters in AI
RL agents cannot see future rewards directly — they must estimate long-term return from the current moment. The value function compresses this into a single scalar: V(s) for state value, Q(s,a) for action value. Temporal difference learning updates this estimate efficiently without needing full trajectories. Every major RL algorithm — Q-learning, PPO, A3C, AlphaGo — relies on a value function to guide policy improvement.
Key Points
| Aspect | Description |
|---|---|
| V(s) | State-value function — expected return from state s following policy π |
| Critic | Neural network that approximates V(s) or Q(s,a) in actor-critic methods |
| Q(s, a) | Action-value function — expected return from taking action a in state s, then following π |
| TD error | δ = R + γ·V(s') − V(s) — the learning signal used to update value estimates |
| Advantage A(s,a) | A(s,a) = Q(s,a) − V(s) — how much better action a is vs the average action |
| Bellman equation | V(s) = R + γ · V(s') — recursive definition enabling bootstrapped updates |
Simple Analogy
A chess player's intuitive board evaluation: without computing every possible future move, an experienced player looks at a position and knows roughly how good it is — that gut feeling is the value function. It encodes millions of games of experience into a single "this position is winning/losing" signal.
Common Usage Examples
- Q-learning:
Q[s,a] += alpha * (reward + gamma * max(Q[s_next]) - Q[s,a]) ValueNetwork(nn.Module)— critic in PPO/A3C outputs scalar V(s)- DQN:
nn.Linear(hidden, n_actions)— output head gives Q(s, a) for all actions - Advantage estimation:
advantage = returns - values.detach()— in PPO training loop stable_baselines3.PPO— trains actor (policy) and critic (value) simultaneously
Summary
In short: The value function is RL's crystal ball — a learned estimate of future reward from any state that enables agents to make decisions without enumerating all future consequences.