Blind search tries directions at random. If instead you can work out which way improvement lies, you can climb far faster. That direction is the policy gradient.
Policy search in Chapter 9 moved by trial. A sharper approach uses gradient information: the direction in which nudging each policy parameter would most increase expected return. Follow that uphill and you optimise efficiently. But there's a catch — you can't differentiate a project you can only simulate and observe. Policy gradient estimation is the set of techniques for estimating that uphill direction from sampled outcomes alone.
1Estimating a direction you can't compute
Several estimators exist. Finite differences wiggle each parameter a little and measure the change in return — simple but slow and noisy. The likelihood-ratio estimator is the elegant workhorse: a mathematical identity lets you estimate the gradient purely from sampled trajectories and the policy's own probabilities, with no model of the environment required. You run the policy, record what happened, and the formula tells you how to shift the parameters to make good outcomes more likely.
2The enemy is variance
These estimates are unbiased but noisy — project outcomes vary wildly from run to run, so a raw gradient estimate can point in a wildly different direction each time. Left unchecked, that noise makes learning crawl or wander. Two standard fixes sharpen the signal dramatically:
- Reward-to-go credits each decision only with the rewards that came after it, not the whole trajectory's return. A decision shouldn't be praised or blamed for outcomes that preceded it.
- Baseline subtraction compares each outcome against a reference level (typically the expected return) and updates on the difference. Doing better than expected pushes the parameters one way; worse, the other. It shrinks the noise without biasing the direction.
You can estimate which way to improve a response rule from observed outcomes alone, no model needed — but only if you control the noise. Reward-to-go and baselines are what turn a jittery estimate into a dependable direction.
When you tune a response strategy against outcomes, resist over-reacting to any single project — one result is one noisy sample of a variable process. Judge each decision by what followed it, not by the whole project's fortunes, and always against a baseline of what you'd normally expect. That is precisely how to tell a genuine improvement signal from ordinary run-to-run variation — the difference between learning and chasing noise.