← AI Terminology
Contextual Bandit
A contextual bandit chooses actions based on observed context features each round, receives a reward for the chosen action only, and never sees counterfactual rewards for other actions.
It sits between multi-armed bandits and full RL.
It sits between multi-armed bandits and full RL.
Why It Matters in AI
Personalisation problems (which article/ad/UI for this user now) are often contextual bandits, not full MDPs. They power large-scale ranking and experimentation systems with partial feedback.
Key Points
| Aspect | Description |
|---|---|
| Use | Recommendations, ads, clinical assignment |
| Vs RL | Usually no long-horizon state evolution modelled |
| Context | Features x_t each round |
| Related | Multi-armed bandit, off-policy evaluation |
| Feedback | Only reward for chosen arm |
| Algorithms | LinUCB, contextual Thompson, offline IPS learners |
Simple Analogy
A waiter who sees each customer (context) and picks one dish to recommend, learning only from whether that customer was happy — not from the dishes not served.
Common Usage Examples
- LinUCB news recommendation
- Off-policy eval with inverse propensity scores
- Personalise email subject lines
- Log propensities for later learning
Summary
In short: Contextual bandits pick actions from features with partial feedback — the standard model for personalised decisions without full RL.