Part II has run two threads — estimating the value of situations, and directly tuning the decision rule. Actor–critic methods weave them into one.
Value-based methods (Chapters 6–8) ask "how good is this state?" and act greedily. Policy methods (Chapters 9–11) tune the decision rule directly but lean on noisy outcome signals. Actor–critic methods take the best of both: a critic that learns to judge situations, feeding a lower-noise signal to an actor that improves the decision rule. It is the synthesis the whole part has been building toward, and the backbone of modern reinforcement learning.
1Two roles, one loop
The actor is the policy — it chooses actions. The critic is a value function — it estimates how good the current situation and the actor's choices are. Instead of judging each decision against the raw, noisy final return (as bare policy gradients do), the actor is updated using the critic's assessment, which is far steadier. In turn, the critic learns from what actually happens as the actor acts. Each improves because the other does.
2The advantage signal, and refinements
The critic's most useful output is the advantage: how much better (or worse) an action turned out than the critic expected. Positive advantage reinforces the choice; negative discourages it. Generalized advantage estimation tunes the trade-off between a low-noise but slightly biased advantage and a high-noise but unbiased one. A deterministic policy gradient variant suits continuous actions, and pairing an actor–critic with the tree search of Chapter 8 produces the class of systems that mastered games like Go — a learned critic guiding the search, the search improving the actor.
An actor decides and a critic judges. Because the actor learns from the critic's steady assessment rather than from raw noisy outcomes, the pair improves faster and more reliably than either approach alone.
The most effective risk operating models pair the same two roles: a forecasting capability that continuously assesses how healthy each project is, and a decision function that acts on those assessments — with each feeding the other. The forecaster tells the decision-maker not just the outcome but whether it beat expectations; the decision-maker's actions give the forecaster fresh evidence to sharpen its judgement. Neither the dashboard nor the playbook works nearly as well in isolation.