← AI Terminology
RLHF - Reinforcement Learning from Human Feedback
RLHF (Reinforcement Learning from Human Feedback) is a training technique that uses human preference judgements to train a reward model, then applies reinforcement learning (PPO) to fine-tune an LLM to produce outputs that maximise that reward — aligning model behaviour with human values and preferences.
It is the key alignment technique behind ChatGPT, Claude, and every frontier assistant model.
It is the key alignment technique behind ChatGPT, Claude, and every frontier assistant model.
Why It Matters in AI
Instruction tuning (SFT) teaches a model to follow commands, but doesn't teach it to be helpful, honest, and harmless by human standards. RLHF fills this gap: human raters compare model outputs and indicate which is better, a reward model learns these preferences, and PPO fine-tunes the LLM to maximise the reward. InstructGPT (2022) showed RLHF-trained models were dramatically preferred by humans over 100× larger SFT-only models. It is the transformation that made LLMs safe and useful at scale.
Key Points
| Aspect | Description |
|---|---|
| KL penalty | Prevents the policy from drifting too far from SFT — avoids reward hacking |
| Phase 2: RM | Reward model trained on human preference rankings — learns to score response quality |
| Alternatives | DPO, ORPO, RLOO — replace PPO complexity with simpler contrastive objectives; increasingly preferred |
| Phase 1: SFT | Supervised fine-tuning on high-quality instruction-response pairs — establishes baseline policy |
| Phase 3: PPO | Policy (LLM) fine-tuned with PPO to maximise reward model score + KL penalty vs SFT baseline |
| Reward hacking | Model learns to exploit reward model flaws rather than being genuinely helpful — key failure mode |
Simple Analogy
A language school that grades students not on grammar tests (SFT) but by asking native speakers to compare pairs of student essays and vote which is better (human preference). The reward model learns the graders' aesthetic; the PPO training is the student practising to improve their grade. Over millions of such comparisons, the student learns to write like a fluent, helpful human.
Common Usage Examples
- InstructGPT: SFT on 13K examples + RM trained on 33K comparisons + PPO fine-tuning — blueprint for ChatGPT
from trl import PPOTrainer, RewardTrainer— HuggingFace TRL implements full RLHF pipeline- Reward model:
AutoModelForSequenceClassification.from_pretrained("Llama-3-8B-reward")— scalar output - Constitutional AI (Anthropic): RL with AI feedback (RLAIF) — replace human raters with Claude
PPOConfig(kl_penalty="kl", init_kl_coef=0.2, target_kl=6.0)— KL penalty hyperparameters
Summary
In short: RLHF trains LLMs to be helpful, honest, and harmless by using human preference rankings to build a reward model, then applying reinforcement learning to optimise the LLM against that reward — the alignment technique that transformed raw language models into safe, useful assistants.