← AI Terminology
PPO - Proximal Policy Optimization
PPO (Proximal Policy Optimization) is a reinforcement learning algorithm that optimises a policy by taking gradient steps that are constrained to stay close to the previous policy — using a clipped probability ratio objective to prevent destabilisingly large policy updates.
It is the primary algorithm used for RLHF alignment training of frontier LLMs.
It is the primary algorithm used for RLHF alignment training of frontier LLMs.
Why It Matters in AI
Earlier policy gradient methods either took too-large steps (destabilising training) or were computationally expensive to constrain (TRPO). PPO solved both: a simple clipping trick limits how much the policy can change per update, making training stable with standard SGD. This simplicity and stability made PPO the de-facto RL algorithm — used for OpenAI Five (Dota 2), robotics, and crucially for RLHF in ChatGPT, GPT-4, Claude, and Gemini alignment training.
Key Points
| Aspect | Description |
|---|---|
| Clipping | clip(r, 1-ε, 1+ε) — restrict ratio to [0.8, 1.2] with ε=0.2; no gradient from outside range |
| Objective | L_CLIP = E[min(r × Â, clip(r, 1-ε, 1+ε) × Â)] — Â is advantage estimate |
| KL penalty | In RLHF: additional KL divergence penalty vs SFT reference model — prevents reward hacking |
| Ratio r(θ) | r = π_new(a |
| Actor-critic | PPO pairs a policy network (actor) with a value network (critic) for advantage estimation |
| RLHF pipeline | Reward model scores response quality → PPO fine-tunes LLM policy to maximise reward + KL term |
Simple Analogy
A business adjusting its strategy: you can improve, but each quarterly change must stay within ±20% of last quarter's strategy (clipping). This prevents panic pivots that destroy what worked before — improving steadily while staying close to proven strategies.
Common Usage Examples
from trl import PPOTrainer, PPOConfig; trainer = PPOTrainer(config, model, ref_model, tokenizer, reward_model)trainer.step(queries, responses, rewards)— one PPO update from LLM-generated response batchesPPOConfig(kl_penalty="kl", init_kl_coef=0.2)— KL penalty coefficient to reference model- OpenAI Five: PPO on 45,000 years of game experience over 10 months — defeated world champion Dota 2 team
- ClipPPO objective:
torch.min(ratio * advantage, torch.clamp(ratio, 0.8, 1.2) * advantage).mean()
Summary
In short: PPO stabilises reinforcement learning by clipping policy updates to prevent destabilising changes — the algorithm powering RLHF alignment training for every major frontier LLM, from ChatGPT to Claude.