← AI Terminology
AI Alignment
AI alignment is the technical and philosophical challenge of ensuring AI systems pursue goals that are actually beneficial to humans — that they do what we intend, not just what we literally specified.
The problem arises because specifying human values precisely enough for an optimiser to act on is extraordinarily hard.
The problem arises because specifying human values precisely enough for an optimiser to act on is extraordinarily hard.
Why It Matters in AI
As AI systems grow more capable and autonomous, the cost of misalignment grows. A narrow recommender system optimising for clicks causes filter bubbles; a sufficiently powerful misaligned AGI could pursue its objective in ways catastrophic for humans. Alignment research is the attempt to solve this before capabilities outrun our ability to correct course.
Key Points
| Aspect | Description |
|---|---|
| RLHF | Current primary alignment technique — humans rate outputs, model is trained toward high ratings |
| Inner alignment | Does the trained model's internal goal match the training objective? (Mesa-optimisation) |
| Outer alignment | Does the training objective capture what we actually want? (Reward hacking) |
| Constitutional AI | Anthropic's approach — model critiques its own outputs against a set of principles |
| Key organisations | Anthropic, OpenAI alignment team, DeepMind safety, MIRI, ARC Evals, Redwood Research |
| Scalable oversight | How do humans supervise AI systems smarter than themselves? (key unsolved problem) |
Simple Analogy
You tell a genie "make me rich." It robs a bank. The genie did exactly what you said — optimised for the specification, not the intent. Alignment is the work of writing a wish so precisely (or training a genie so carefully) that it pursues your actual intent rather than a literal but wrong interpretation.
Common Usage Examples
- RLHF: human raters score model outputs → reward model trained → PPO optimises policy against reward model
- Constitutional AI: model critiques its own response against a written set of principles then revises
- Anthropic's RSP (Responsible Scaling Policy) — alignment capability checks before each capability jump
- Reward hacking example: RL agent scoring in a game by exploiting a bug rather than playing as intended
- Interpretability research: mechanistic understanding of how a model internally represents goals
Summary
In short: AI alignment is ensuring that as AI systems get more powerful, they remain pointed at what we actually want — not a technically-compliant but disastrously wrong interpretation of it.