← AI Terminology
Multi-Armed Bandit
A multi-armed bandit is a sequential decision problem: choose among actions (“arms”) with unknown reward distributions to maximise cumulative reward, balancing exploration and exploitation.
It is the simplest reinforcement setting and a production staple for online optimisation.
It is the simplest reinforcement setting and a production staple for online optimisation.
Why It Matters in AI
A/B tests are batchy; bandits adapt allocation toward better arms sooner. Used in ranking, ads, recommendations, and adaptive experiment design — core decision-making under uncertainty.
Key Points
| Aspect | Description |
|---|---|
| Use | Online UI optimisation, ad creative selection |
| Metric | Regret vs always picking the best arm |
| Problem | Unknown reward means; pull arms over time |
| Related | Contextual bandits, exploration-exploitation |
| Algorithms | ε-greedy, UCB, Thompson sampling |
| Vs full RL | No long state horizon; usually stateless |
Simple Analogy
A row of casino slot machines: try unfamiliar levers enough to learn, but spend more time on the ones that pay.
Common Usage Examples
- Thompson sampling for button colours
- UCB news article recommendation
- Adaptive clinical dose-finding analogies
- Simulate regret curves
Summary
In short: Multi-armed bandits sequentially choose actions to maximise reward under uncertainty — the workhorse of adaptive online experiments.