Belief-State Planning: Offline, Online and Controllers

From filtering to planning

Filtering estimates the hidden state; planning decides what to do with that uncertainty.

A partially observable decision problem adds observations to the state-transition and reward model. After each action, the system moves to a hidden state, emits an observation and updates the belief. The belief therefore evolves stochastically according to both the action and future observation.

A policy maps beliefs to actions. The same physical observation can lead to different actions depending on prior belief, because its meaning depends on context. This is why “if sensor > threshold, repair” can be inferior to a belief-based policy when measurement quality and degradation history matter.

The belief MDP

A partially observable problem can be transformed conceptually into a fully observable MDP whose state is the belief distribution.

The immediate reward at a belief is the probability-weighted reward across hidden states. The belief transition accounts for every possible observation after an action. This transformation is mathematically powerful but computationally challenging because the belief space is continuous and high-dimensional.

The optimal value balances direct task reward and information value automatically. An inspection action can be chosen because it changes the future belief and enables better later actions, even if inspection has a short-term cost.

Exact belief-state planning

For small finite problems, value functions can be represented with structured components over belief space.

Finite-horizon value functions have a piecewise-linear structure under standard assumptions. Dynamic-programming backups generate candidate value vectors associated with conditional plans. The number of vectors can grow rapidly as horizon, actions and observations increase, creating the classic computational difficulty of exact partially observable planning.

Exact methods are useful for small benchmark or safety-critical subproblems where the state, action and observation sets are deliberately compact. For larger business systems, approximation is usually necessary.

Offline approximate planning

Offline methods invest computation before deployment to construct a policy or approximate value over relevant beliefs.

Point-based methods focus updates on a set of sampled or reachable beliefs instead of the entire continuous belief simplex. The logic is pragmatic: an operational system will visit only a subset of mathematically possible beliefs. Sampling that region can yield strong policies with far less computation.

Quality depends on belief coverage. If deployment reaches beliefs not represented during planning, the policy may extrapolate poorly. Generate belief points through realistic simulations, stress scenarios and exploratory policies rather than only nominal trajectories.

Online belief-space planning

Online methods build a look-ahead tree from the current belief at decision time.

Each action branch is followed by possible observations, each of which creates a new posterior belief. Because observation branching can be large or continuous, search must use sampling, bounds and heuristics. The advantage is focus: computation is spent on the current decision rather than every possible belief.

Online planning is particularly useful when a good simulator exists and decisions are not too time constrained. Maintain a safe fallback policy for computation failure.

Controller abstractions

Finite-state controllers compress a history-dependent policy into a small internal controller state.

A controller node selects an action. After an observation, the controller transitions to another node. The internal controller state therefore summarises relevant history without explicitly maintaining a full high-dimensional belief in the deployed rule.

Controller abstractions can be easier to implement, verify and explain. The trade-off is representational limitation: a small controller may not capture fine distinctions in uncertainty. Search or optimisation can choose controller parameters while validation tests whether the compact policy loses material value.

Information actions and dual purpose

Partially observable planning makes the value of sensing explicit.

An action can both affect the physical system and reveal information. A low-speed diagnostic run may reduce production now but generate a cleaner signal about machine health. A small market experiment may produce some sales while mainly learning demand. These are dual-purpose actions.

The planner should model observation quality and timing. More information is not automatically better if it costs too much, arrives too late or changes the system adversely. This links belief-state planning directly to value-of-information analysis.

Worked example: inspection and repair

A generic asset can be healthy, degraded or failed, but only noisy inspection results are observed.

The policy may continue operation when the belief is strongly healthy, inspect when uncertainty is high, repair when degradation probability is high and replace after failure. A simple threshold on the latest inspection cannot represent the distinction between a surprising bad result in an otherwise healthy history and the same bad result after several warning signs.

Offline planning can compute a compact rule across likely beliefs; online planning can search more deeply from unusual beliefs. A finite-state controller can provide an implementable approximation for technicians or embedded control logic.

Implementation controls

Belief-state systems combine model uncertainty, estimator error and planning approximation.

Belief-policy handover to users

A technically correct method still needs an auditable operating translation.

When a belief-based policy is used by people, the recommendation should be accompanied by a concise explanation of the uncertainty driving it. For example, “inspect” can be explained as worthwhile because the current evidence leaves two materially different hidden states plausible and the next action would change depending on the result. This helps users distinguish information-seeking actions from hesitation. Define how manual observations enter the belief update, how stale information expires and how overrides are recorded. Those records are valuable for recalibration because they reveal situations in which the model’s observation or transition assumptions did not match operational judgement.

Application checklist

• Use belief as the policy state when hidden-state uncertainty affects action value.
• Model observation timing and quality as part of the decision problem.
• Recognise information-gathering actions as legitimate choices.
• Use exact planning only for sufficiently small partially observable models.
• Cover reachable and stressed beliefs in offline approximations.
• Use online search when current-state focus and computation time permit it.
• Consider compact controllers where implementation simplicity matters.
• Validate transition, observation, filtering and planning layers separately and together.