← AI Terminology
Exploration vs Exploitation
Exploration vs exploitation is the fundamental dilemma in reinforcement learning and decision-making: should an agent try new actions to discover potentially better rewards (explore), or stick with the best action it knows so far to maximise current reward (exploit)?
Resolving this tradeoff is central to every RL algorithm and many real-world AI systems.
Resolving this tradeoff is central to every RL algorithm and many real-world AI systems.
Why It Matters in AI
An agent that only exploits will miss better strategies; one that only explores wastes time on known-bad actions. The tradeoff appears everywhere: a recommendation system that always recommends popular items (exploitation) never discovers what a user might prefer (exploration). Drug discovery, A/B testing, hyperparameter search, and game-playing agents all face this dilemma.
Key Points
| Aspect | Description |
|---|---|
| UCB | Upper Confidence Bound — prefer actions with high estimated value OR high uncertainty |
| ε decay | Start with high ε (lots of exploration), decay toward 0 as the agent gains experience |
| ε-greedy | With probability ε, take a random action (explore); otherwise take the best known (exploit) |
| Bandit problem | Simplified RL version (no state transitions) — canonical setting for exploring arm choices |
| Curiosity-driven | Intrinsic motivation: reward the agent for visiting novel states regardless of extrinsic reward |
| Thompson sampling | Sample from posterior over action values — naturally balances explore/exploit probabilistically |
Simple Analogy
Choosing a restaurant: exploit means going to your favourite every time (known good); explore means trying a new place (might be better or worse). If you never explore, you'll never discover your new favourite. If you always explore, you never enjoy a reliable great meal. The optimal strategy adapts: explore more when you're new to town, exploit more once you've found the good spots.
Common Usage Examples
- ε-greedy in DQN: ε starts at 1.0, decays linearly to 0.1 over 1M steps
- UCB1 bandit:
action = argmax(Q(a) + c * sqrt(log(t) / N(a)))— prefers uncertain actions - Thompson sampling: used in LinkedIn/Bing ad serving for real-time exploration
- Curiosity-driven RL: Random Network Distillation (RND) reward for visiting novel states
- A/B testing: multi-armed bandit replaces fixed allocation — exploration stops early when winner clear
Summary
In short: Exploration vs exploitation is whether to try something new or stick with what works — the central tension in RL and any learning system that must act before it has complete information.