← AI Terminology
Q-Learning
Q-learning is a model-free reinforcement learning algorithm that learns the value of taking an action in a given state (the Q-function) by iteratively updating Q(s,a) estimates using the Bellman equation — without requiring a model of the environment's transition dynamics.
It is the foundational value-based RL algorithm, and the basis for Deep Q-Networks (DQN).
It is the foundational value-based RL algorithm, and the basis for Deep Q-Networks (DQN).
Why It Matters in AI
Q-learning was among the first RL algorithms proven to converge to the optimal policy for discrete action spaces (under tabular conditions). DeepQ-Networks (DQN, 2013) combined Q-learning with deep neural networks and replay buffers to play Atari games at superhuman level — the breakthrough that launched the modern deep RL era. Understanding Q-learning is the entry point to all value-based RL methods: DDQN, Dueling DQN, Rainbow, TD3.
Key Points
| Aspect | Description |
|---|---|
| DQN | Neural network approximates Q(s,a) — experience replay + target network for stability |
| Off-policy | Learns about the optimal policy while following any exploration policy (ε-greedy) |
| Q-function | Q(s,a): expected cumulative discounted reward from state s, taking action a, then acting optimally |
| Convergence | Tabular Q-learning: guaranteed to converge to Q* if all state-action pairs visited infinitely often |
| Tabular limit | Q-table works for small state spaces; neural network required for large/continuous state spaces |
| Bellman update | Q(s,a) ← Q(s,a) + α[r + γ × max_a' Q(s',a') − Q(s,a)] — temporal difference target |
Simple Analogy
A tourist building a mental map of which restaurants to visit: after eating at each combination of neighbourhood + restaurant type, they update their score for "going to an Italian place in Montmartre." Over time, revisiting many combinations, their ratings converge to reflect the true quality of each choice — that accumulated score table is the Q-function.
Common Usage Examples
- Tabular:
Q[state][action] += alpha * (reward + gamma * max(Q[next_state]) - Q[state][action]) - DQN:
from stable_baselines3 import DQN; model = DQN("MlpPolicy", env, verbose=1); model.learn(10000) - OpenAI Gym CartPole: Q-learning agent learns to balance pole in ~500 episodes
- ε-greedy exploration:
action = argmax(Q[s]) if rand() > ε else random_action() - Rainbow DQN: combines 6 improvements (PER, dueling, double Q, distributional, noisy, n-step)
Summary
In short: Q-learning learns the value of each state-action pair by iteratively applying the Bellman equation — the foundational value-based RL algorithm that, combined with neural networks in DQN, achieved superhuman Atari game performance and launched deep RL.