There's a shortcut past trial-and-error: learn from someone who already knows. But copying an expert is subtler than it looks, and the way it fails is deeply instructive.
When a skilled practitioner already makes good decisions, you don't need to rediscover good behaviour through costly exploration — you can learn it from their example. Imitation learning builds a policy from demonstrations rather than from a reward signal or a model. It's the formal version of apprenticeship, and it's how you might turn the hard-won judgement of your best project and risk experts into a reusable, teachable policy.
1Behavioural cloning, and its hidden trap
The obvious approach, behavioural cloning, treats imitation as ordinary supervised learning: collect examples of the situations the expert faced and the actions they took, then fit a policy that reproduces those actions. Simple and often a fine start. But it hides a serious flaw. The learner is only ever trained on the situations the expert visited — and experts, being good, mostly stay on well-managed paths. The moment the learner makes a small error and drifts somewhere the expert never went, it has no guidance, so it errs again, drifting further into unfamiliar territory. Small mistakes compound into large ones. This is distribution shift, and it is the central difficulty of imitation.
2Two better answers
- Query the expert where the learner actually goes. Instead of a fixed set of demonstrations, iteratively run the learner, note the situations it drifts into, and ask the expert what they would do there — then add those to the training set. Over rounds, the learner gets guidance precisely for the situations it's prone to reach, and the compounding-error problem is tamed.
- Infer the goal, not the moves. Inverse reinforcement learning asks a deeper question: rather than copying the expert's actions, work out what objective the expert appears to be pursuing, then optimise that. Recovering the underlying intent generalises far better than mimicking surface behaviour — a policy that understands the goal can handle situations the demonstrations never showed, because it knows what it's trying to achieve.
Copying what an expert does breaks down the instant you leave the paths they walked. Capturing what the expert is trying to achieve survives into new situations — which is the difference between a brittle checklist and genuine transferred judgement.
Part III faced an unknown model: whether to explore or exploit, and whether to learn the model (model-based), learn actions directly (model-free), or learn from an expert (imitation). But every method in Parts I–III assumed you could at least see the current state — you knew which situation you were in. Real projects deny even that: status is reported late, partially, and sometimes wrongly. Part IV takes on the hardest case — deciding well when you cannot directly observe the true state at all.
To capture expert judgement, start by recording what your best practitioners do — but know that a demonstration set alone yields a brittle policy that fails in exactly the off-script situations where judgement matters most. Strengthen it by asking your experts specifically about the awkward, unfamiliar cases a junior would stumble into, and by drawing out the objectives behind their choices, not just the choices. The most transferable thing you can extract from an expert is not their moves but their aims.