← AI Terminology
Policy Gradient
Policy gradient is a class of reinforcement learning algorithms that directly optimise the policy — the function mapping states to actions — by computing the gradient of expected cumulative reward with respect to policy parameters and ascending it via gradient descent.
It is the RL foundation behind PPO, which powers RLHF for aligning LLMs.
It is the RL foundation behind PPO, which powers RLHF for aligning LLMs.
Why It Matters in AI
Unlike value-based RL (Q-learning), which learns a value function and derives a policy, policy gradient methods directly parameterise and optimise the policy — making them suitable for continuous action spaces and stochastic policies. REINFORCE, A2C, A3C, PPO, and TRPO are all policy gradient methods. PPO's application to RLHF is how GPT-4, Claude, and Gemini are aligned with human preferences — making policy gradient the mathematical engine behind modern AI alignment.
Key Points
| Aspect | Description |
|---|---|
| PPO | Proximal Policy Optimization: clips policy update ratio to prevent too-large steps — stable |
| RLHF | Reward model provides scalar reward; PPO optimises policy (LLM) to maximise reward + KL penalty |
| TRPO | Trust Region Policy Optimization: constrain KL divergence between old and new policy |
| Baseline | Subtract a baseline (e.g. value function) from G to reduce variance — actor-critic approach |
| REINFORCE | Monte Carlo policy gradient: ∇J(θ) = E[∇ log π(a |
| Actor-critic | Actor = policy network; Critic = value network; critic provides low-variance baseline for actor |
Simple Analogy
A slot machine player adjusting their betting strategy: they try different strategies (actions), observe payouts (rewards), and increase the probability of strategies that paid well. Policy gradient is exactly this — but applied mathematically: raise the probability of actions that led to high reward, lower those that led to low reward.
Common Usage Examples
- REINFORCE:
loss = -log_prob * reward; loss.backward()— simplest policy gradient update - PPO:
from trl import PPOTrainer; trainer = PPOTrainer(config, model, ref_model, reward_model) torch.distributions.Categorical(probs).log_prob(action)— log probability of a discrete action- OpenAI Gym: policy gradient on CartPole — agent learns to balance by maximising episode length reward
- RLHF pipeline: PPO policy gradient with KL penalty:
reward = reward_model(response) - β × KL(policy || ref)
Summary
In short: Policy gradient directly optimises a policy by ascending the gradient of expected reward — the RL foundation underlying PPO, which is the algorithm used to align frontier LLMs like GPT-4 and Claude with human preferences.