A direction is only half the decision; the other half is how far to step. On a noisy signal, an over-confident stride can undo months of careful improvement in one move.
Chapter 10 gave us the uphill direction. Policy gradient optimization is about using it responsibly. The naïve approach — gradient ascent, taking a step proportional to the estimated gradient — works when steps are small and the signal is clean, but real gradient estimates are noisy and the return surface is treacherous. Too large a step, informed by a misleading estimate, can send a good policy off a cliff into far worse behaviour. The methods here exist to make improvement monotonic and safe.
1Respecting the geometry: the natural gradient
A plain gradient treats every parameter as if a unit change means the same thing everywhere. It doesn't — a small parameter tweak can barely alter a policy in one region and completely transform it in another. The natural gradient corrects for this by measuring change in terms of how much the policy's behaviour actually shifts, not how much its numbers move. The result is steadier progress that doesn't accidentally lurch just because the parameterisation was uneven.
2Fencing the step: trust regions
The central safeguard is the trust region: explicitly cap how much the policy is allowed to change in a single update, so you never step beyond the neighbourhood where your gradient estimate is trustworthy. Take the best step you can within that fence, then move the fence and repeat. Trust-region methods and their lighter-weight cousin — a clipped surrogate objective that discourages the update from straying too far from the current policy — are the workhorses of modern reliable policy improvement.
Safe improvement means bounded improvement. Limit how far a policy can move on any one update — to the region where your evidence is trustworthy — and you climb reliably instead of gambling the whole strategy on a noisy step.
Don't overhaul a working risk-response strategy in one dramatic swing because a single quarter's data pointed somewhere. Make bounded, defensible adjustments that stay close to what's already proven, and let evidence accumulate before you move further. "Improve, but only within limits we can justify" is not timidity — it's the discipline that keeps continuous improvement from periodically blowing up. This is course-correction, not reinvention.