← AI Terminology
Reward Model
A reward model is a neural network trained on human preference data to predict a scalar quality score for any LLM-generated response — providing the reward signal for reinforcement learning in the RLHF pipeline that aligns language models with human values.
It replaces direct human rating at scale during RL fine-tuning.
It replaces direct human rating at scale during RL fine-tuning.
Why It Matters in AI
RLHF requires thousands of reward signals per second during PPO training — impossible with real human raters. The reward model is trained once on human preference comparisons and then provides instant scalar scores to guide PPO. Its quality is critical: a poorly trained reward model is exploited by the policy (reward hacking), leading to a model that scores high on the reward model but is actually less helpful. Reward model quality is a core determinant of final model alignment quality.
Key Points
| Aspect | Description |
|---|---|
| KL penalty | Added to RL objective to constrain policy deviation from SFT — limits how much RM can be exploited |
| Alternatives | DPO, ORPO bypass RM entirely — directly optimise policy on preference data without RL |
| Architecture | LLM + scalar head — replace LM head with a single linear layer that outputs one number |
| Training data | Pairs (prompt, chosen_response, rejected_response) — rated by human annotators |
| Training loss | loss = -log(sigmoid(reward_chosen - reward_rejected)) — Bradley-Terry pairwise ranking loss |
| Reward hacking | Policy exploits RM weaknesses — generate text that scores high but isn't actually good |
Simple Analogy
A trained film critic who watched thousands of paired movies and rated which was better: now they can instantly score any new film without waiting for audience response. The RLHF system sends the LLM's outputs to this critic for instant quality scores — guiding the LLM to produce Oscar-worthy responses at scale.
Common Usage Examples
from trl import RewardTrainer, RewardConfig; trainer = RewardTrainer(model, train_dataset=pairs_dataset)- Reward model inference:
score = reward_model(input_ids).logits[0]— scalar quality score reward_model = AutoModelForSequenceClassification.from_pretrained("meta-llama/Llama-3-8B", num_labels=1)- HuggingFace Hub:
OpenAssistant/reward-model-deberta-v3-large-v2— open-source reward model - PPO reward:
rewards = reward_model(responses) - β * kl_div(policy, reference_policy)— adjusted reward
Summary
In short: A reward model is a neural network trained on human preference comparisons that provides instant quality scores during RLHF — the learned proxy for human judgment that guides PPO to align LLMs with human values at scale.