← AI Terminology
Outcome Reward Model
An outcome reward model (ORM) scores only the final result of a trajectory (answer correct/incorrect, preference win/lose), not intermediate steps.
It is the classic reward model form used in RLHF and many best-of-N setups.
It is the classic reward model form used in RLHF and many best-of-N setups.
Why It Matters in AI
Preference learning and RLHF typically need a scalar for a whole response. ORMs are simpler to train than PRMs but provide sparse signal on long reasoning. Choosing ORM vs PRM shapes alignment and reasoning pipelines.
Key Points
| Aspect | Description |
|---|---|
| Use | RLHF, rejection sampling, best-of-N reranking |
| Data | Human pairwise preferences or verifiable final answers |
| Input | Prompt + full completion (sometimes with reference) |
| Output | Scalar score or preference probability |
| Related | Bradley-Terry RMs, DPO (implicit reward) |
| Sparse credit | Hard to assign blame inside long chains |
Simple Analogy
A judge who only sees the final exam score, not the working — simple to grade, little feedback on which steps went wrong.
Common Usage Examples
- Classic RLHF reward models
- Best-of-N: generate N, pick highest ORM score
- Unit-test pass/fail as outcome reward for code
- Compare ORM vs PRM on MATH search
Summary
In short: An outcome reward model scores only the final answer or full response — simple and common in RLHF, but sparse for long reasoning.