← AI Terminology
Preference Data
Preference data records which among candidate responses is better (pairwise or ranked), used to train reward models or direct preference methods like DPO.
It is the fuel of modern alignment beyond pure SFT.
It is the fuel of modern alignment beyond pure SFT.
Why It Matters in AI
SFT shows one good answer; preferences teach relative quality and subtle policy. Collecting, cleaning, and balancing preference data dominates RLHF/DPO quality. Garbage prefs → sycophantic or hacked models.
Key Points
| Aspect | Description |
|---|---|
| Use | RM training, DPO/ORPO/KTO, best-of-N targets |
| Bias | Length, style, annotator subjectivity |
| Form | Prompt + chosen + rejected (+ ties) |
| Scale | Thousands to millions of pairs |
| Related | RLHF, DPO, RLAIF |
| Sources | Humans, RLAIF judges, hybrid |
Simple Analogy
Not only sample essays, but marked pairs: “A is better than B for this prompt” — teaching taste, not only imitation.
Common Usage Examples
- Anthropic HH-RLHF style datasets
- Train DPO on chosen/rejected columns
- Audit length bias in prefs
- Inter-annotator agreement checks
Summary
In short: Preference data labels which outputs are better — the supervision behind RLHF, DPO, and modern alignment.