← AI Terminology
Actor-Critic
Actor-Critic is a reinforcement learning architecture that trains two components simultaneously: an actor that selects actions and a critic that evaluates how good those actions are.
The critic's value estimates guide the actor's learning, reducing the high variance that pure policy-gradient methods suffer from.
The critic's value estimates guide the actor's learning, reducing the high variance that pure policy-gradient methods suffer from.
Why It Matters in AI
Actor-Critic is the backbone of the most successful modern RL algorithms — PPO, SAC, A3C, and TD3 all use this structure. It hits a sweet spot between Monte Carlo methods (high variance) and pure value-based methods (off-policy instability), making it the go-to framework for continuous-action tasks and RLHF for LLMs.
Key Points
| Aspect | Description |
|---|---|
| Actor | Policy network π(a |
| Critic | Value network V(s) or Q(s,a) — estimates expected future reward from a state or state-action |
| Advantage | A(s,a) = Q(s,a) − V(s) — measures how much better an action is than average; reduces variance |
| Frameworks | Stable-Baselines3, RLlib, TRL (HuggingFace) for RLHF |
| Key variants | A2C (synchronous), A3C (asynchronous), PPO (clipped), SAC (entropy-regularised) |
| Role in RLHF | PPO uses actor-critic to fine-tune LLMs against a reward model signal |
Simple Analogy
Imagine a student (actor) answering exam questions and a tutor (critic) watching and scoring each answer in real time. The student doesn't have to wait until the end of the exam to learn — the tutor's running score shapes every next decision.
Common Usage Examples
- PPO actor-critic used to fine-tune ChatGPT via RLHF (OpenAI, 2022)
stable_baselines3.PPO— actor-critic for continuous control (robotics, games)- A3C training multiple parallel workers to explore different environment states
- SAC (Soft Actor-Critic) for robot locomotion — maximises entropy for exploration
trl.PPOTrainerin HuggingFace for LLM RLHF training
Summary
In short: Actor-Critic combines a policy that acts with a value function that coaches it — the architecture behind most modern RL and LLM alignment systems.