Sometimes you can't build a credible model of the project — it's too novel, too tangled, too unlike anything before. You can still learn what works, straight from experience, with no theory at all.
Model-free methods take the opposite bet to Chapter 15. Instead of learning how the environment works and planning against it, they learn which actions are good directly from the stream of experience — bypassing the model entirely. There's no transition model to estimate, no planning step, and crucially no model error to propagate. You act, you observe how it turned out, and you adjust your sense of which actions pay off. That's it.
1Temporal-difference learning: don't wait for the end
The engine of model-free learning is the temporal-difference update, and its insight is worth dwelling on. You don't wait until a project finishes to judge whether a decision was good. The instant things turn out better or worse than you expected, you nudge your estimate of that situation's value toward the reward you just saw plus your revised estimate of where you've landed. You learn from the surprise, immediately, rather than waiting for the final tally. This "bootstrapping" — updating an estimate using other estimates — is what lets model-free methods learn continuously from an ongoing stream.
2Q-learning and its relatives
Q-learning applies this to action values: it learns, for each situation, how good each available action is, updating those values from observed transitions. It has a useful property — it can learn the value of the best policy even while behaving more cautiously to explore, which matters when experimenting for real carries risk. Its on-policy cousin evaluates the policy you're actually following. For the enormous state spaces of real problems, these values are represented with the function approximation of Chapter 7 — the pairing that produced the well-known deep reinforcement-learning systems.
Model-free learning skips the theory and maps experience straight to "which actions pay off here". It needs more experience than a model-based approach, but it carries no model to be wrong — a fair trade when the environment defies clean modelling.
When a project or programme is too novel or complex to model credibly, you can still improve — by learning directly which responses tend to work out in which situations, accumulated across real outcomes. The temporal-difference habit is the transferable one: revise your read on how a situation is going the moment it beats or misses expectations, instead of withholding judgement until the post-mortem. It is learning from experience without needing a theory of why — slower to converge, but honest about how little you can model.