Part I decided one thing, once. Real projects are a chain of decisions where each choice changes the situation you face next. That needs a different object entirely.
A decision tree can, in principle, unroll a sequence of choices — but it explodes as the horizon lengthens, and it assumes you know exactly which move follows which. Projects don't work that way: you take an action, the world responds partly at random, and you decide again from wherever you land. The Markov decision process (MDP) is the framework built for precisely this loop, and it is the foundation for the whole of Part II.
1The four ingredients
An MDP is defined by four things. A set of states describing the situation you could be in (on track, over budget, blocked on approvals). A set of actions you can take in each. A transition model giving the probability of the next state given the current state and your action — capturing that outcomes are uncertain. And a reward for each state and action, encoding what you value. A discount factor then trades near-term reward against long-term, expressing time preference.
The name comes from the Markov property: the next state depends only on the current state and action, not the entire history. In practice this means the state must be defined richly enough to summarise everything relevant — a modelling discipline in its own right.
2From plan to policy
The solution to an MDP is not a plan — a fixed sequence of moves — but a policy: a rule that prescribes the best action for every state you might reach. This is the crucial mental shift for risk work. A plan says "do A, then B, then C". A policy says "if the project is on track, monitor; if it slips, add resources; if it's badly delayed, rebaseline". It is contingency baked in from the start, an answer ready for every situation rather than a single hopeful path.
2Value, and the Bellman equation
To compare policies you need the value function: the expected discounted reward you'll collect from a given state if you follow a policy from there on. Value obeys a beautifully recursive relationship — the Bellman equation — that says the value of a state is the immediate reward plus the discounted value of wherever you land next. That single recursion is the engine behind every exact method.
3Two ways to solve it exactly
- Policy iteration alternates two steps: evaluate the current policy (compute its value everywhere), then improve it by acting greedily with respect to those values. Repeat. It converges to the optimal policy in surprisingly few rounds.
- Value iteration skips explicit evaluation and simply applies the Bellman optimality update over and over until the values stop changing; the optimal policy falls out at the end.
Both are guaranteed to find the optimal policy when the model is known and the state space is small enough to enumerate. Asynchronous variants update states in any order for efficiency, and the problem can even be cast as a linear program.
An MDP replaces the fragile idea of a plan with the robust idea of a policy — a best response prepared for every state the project could reach. Exact methods compute that policy optimally when you can model and enumerate the situation.
Stop optimising a single expected path and start designing a response rule: for each state your project could be in, what is the right move? Discounting makes your time-preference explicit instead of implicit. The catch — which the rest of Part II exists to address — is that real projects have far too many states to enumerate, and you rarely know the transition model exactly. Everything that follows relaxes one of those two assumptions.