Everything until now assumed you knew the odds. Part III removes that comfort. And the moment you don't know the odds, one tension dominates every choice: exploit, or explore?
Parts I and II handed you a model — transition probabilities, rewards, all given. Real projects rarely do. When you're uncertain how good your options actually are, every decision does double duty: it earns a result and it teaches you something. Exploitation means taking the option you currently believe is best; exploration means trying an uncertain option to learn more, accepting a possibly worse result now for better decisions later. Getting this balance right is the whole of decision-making under an unknown model.
1The bandit: the tension in its purest form
The clean abstraction is the multi-armed bandit: several options, each with an unknown payoff you can only learn by trying it. Pull an arm, get a reward, update your belief. Pour every pull into the arm that looks best so far and you might never discover that a barely-tried option was better all along. Spread your pulls to learn, and you waste them on options you already suspect are poor. The cost of getting it wrong is measured as regret — the gap between what you earned and what the best possible strategy would have earned.
2Strategies for striking the balance
- ε-greedy exploits the best-known option most of the time but explores at random a small fraction of the time. Crude, but effective and trivial to run.
- Optimism under uncertainty (the upper-confidence-bound idea) judges each option not by its average so far but by the top of its plausible range, and picks the highest. Under-tried options carry wide uncertainty, so they get the benefit of the doubt — until enough trials either confirm or deflate them.
- Posterior sampling (Thompson sampling) is elegantly simple: draw one plausible value for each option from your current belief, then act greedily on that draw. Options you're unsure about get sampled optimistically often enough to be tried, and the exploration self-tunes as beliefs sharpen.
When you don't know the odds, the average result so far isn't enough to decide by — you must weigh what an option could be, given how little you've tested it. Give promising unknowns the benefit of the doubt, in proportion to your uncertainty about them.
3This is the pilot-versus-commit decision
Strip away the mathematics and this is the most common strategic question in project delivery. Do you commit to the incumbent supplier, method, or contractor you trust — or run a pilot with the promising newcomer who might be substantially better? Every trial costs time and money and might disappoint; every trial also buys information you can't get any other way. The bandit framing gives that instinct a spine: an untried option deserves a trial when its plausible upside, weighted by how little you know, beats sticking with the proven choice.
Treat "should we try the new approach?" as an explicit explore-exploit call, not a matter of temperament. Don't judge an unproven supplier, tool, or method solely on a thin track record — weigh its plausible upside against your uncertainty, and size the trial accordingly. Run cheap, reversible experiments early while there's time for the information to pay off, and commit hard once the uncertainty has genuinely collapsed. Persisting with the familiar option out of habit is a decision too — usually an expensive one.