If the trouble is that you don't know how the project behaves, the most natural response is to find out — build a model from what you observe, then use everything from Part II to plan against it.
Model-based methods close the gap between "unknown model" and "everything we can do once we have one". You interact with the environment, record what happens, assemble those observations into an estimated model of the dynamics and rewards, and then plan against that learned model using the exact and approximate methods of Part II. As more experience arrives, the model sharpens and the plan improves. It is learning and planning, interleaved.
1Learn, plan, act, repeat
The cycle is straightforward. Act in the environment; observe the transition and reward you received; fold that observation into your model estimate (in the simplest case, tally the observed frequencies of each transition, exactly the parameter learning of Chapter 3); then plan against the current best model to choose what to do next. A Bayesian variant keeps a whole distribution over possible models, which connects directly to exploration — you can plan to visit states precisely because they'd teach you the most about the model.
2Why it's frugal with experience
The great virtue of model-based methods is sample efficiency. Real interactions are expensive — in a project, a single observation might be a whole month of delivery — so you want to squeeze maximum value from each. A learned model lets you do exactly that: every real observation improves the model, and then you can plan (or simulate) against that model as much as you like at low cost. Some approaches deliberately blend a little real experience with a lot of cheap simulated experience drawn from the learned model, getting more improvement per costly interaction.
Don't just react to outcomes — assemble them into a model of how your project actually behaves, then plan against it. A learned model turns each expensive observation into repeated planning value, which is why model-based methods learn fastest when data is scarce.
Early in a project — or the first time you attempt a new kind of project — you don't yet know its dynamics, so build the model as you go: capture how this project actually responds to your actions, and plan against that captured understanding rather than a generic assumption. This is organisational learning made systematic. Because project data is precious and slow to gather, the discipline of extracting a reusable model from every observation, then planning hard against it, is exactly where model-based thinking earns its keep.