Simulation Is an AI Governance Layer, Not Just a Development Tool

Before AI is trusted in consequential operations, leaders need evidence of how it behaves across scenarios it was not optimised to pass.

A model can perform strongly in the environment where it was developed and still behave materially worse when scale, traffic, demand or operating conditions change.

That is not a theoretical concern.

Brittain, Alvarez, Breeden and Jessen's 2022 AAM-Gym research provides a useful example from Advanced Air Mobility. Their testbed was designed to develop, train and validate AI algorithms in representative air-transport scenarios using standardised interfaces, simulation, scenario generation, metrics and analytics.

In one experiment, algorithms trained on a 100-aircraft scenario were evaluated on a more complex 678-aircraft scenario they had not seen during training. All tested algorithms still improved safety relative to the unequipped baseline under the paper's risk-ratio measure, but the risk ratios were materially higher in the larger unseen scenario. The authors suggested additional training or more complex training scenarios might be required.

The lesson is not that those algorithms were unsafe.

It is that performance evidence changes when the operating environment changes.

The Strategic Context

AI governance is often discussed through policies, principles and model documentation.

Those matter, but consequential systems also need an evidence environment.

AAM-Gym is valuable as a source because its architecture makes this environment visible.

The paper's Figure 1 links:

use-case definition

algorithm

standard API

simulation workers

fast-time simulation

scenario generation

metrics

database

playback and analytics

This is more than a software-development pipeline.

It is a repeatable system for asking:

• what problem is the AI solving?
• what inputs and outputs are allowed?
• against what baseline is performance compared?
• which scenarios are used?
• which metrics determine success?
• can experiments be reproduced?
• and how does performance change under different operating conditions?

These are governance questions.

What Leaders Commonly Misread

The first error is to treat training performance as validation.

Training tells us whether the algorithm learned to optimise within the environment it experienced.

Validation asks whether the resulting behaviour is acceptable under conditions relevant to deployment.

The second error is to treat one benchmark as proof of operational quality.

A model can improve average performance while failing in rare but consequential scenarios.

The third error is to compare AI systems without a common baseline.

If different teams use different simulators, scenarios, assumptions and metrics, ranking model performance can become meaningless.

The fourth error is to believe that higher model performance necessarily means better system performance.

In operational settings, leaders also care about:

• alert burden;
• delays;
• throughput;
• holding;
• human workload;
• resilience;
• and failure behaviour.

The fifth error is to assume simulation itself is reality.

Simulation is a model of the operating environment. It can reveal weaknesses and create evidence, but its usefulness depends on fidelity, assumptions and scenario coverage.

Reframing the Issue

Simulation should be reframed as an AI governance layer between development and operational trust.

That layer has at least six purposes.

1. Standardise the decision problem

AAM-Gym defines use cases through objectives, observation spaces and action spaces.

In enterprise AI, the equivalent is clarifying:

• what the system is allowed to observe;
• what decisions it may influence;
• what actions it may take;
• and what outcome it is optimising.

Ambiguous use cases create ambiguous validation.

2. Create a baseline

AAM-Gym can compare an AI system against an unequipped or existing system.

This is essential.

“AI accuracy = 94 per cent” is difficult to interpret without knowing:

• current human performance;
• existing automation performance;
• cost of false positives;
• cost of false negatives;
• and operational trade-offs.

A new system should demonstrate improvement against the relevant baseline, not only against an abstract benchmark.

3. Generate representative scenarios

The paper combines simulated traffic with scenario-generation tools and discusses integration of real-world air-traffic data.

The general governance principle is to create scenarios that represent the actual distribution of operating conditions.

For a healthcare model, that may mean different patient cohorts.

For manufacturing, different equipment states and process disturbances.

For fraud detection, different transaction patterns.

For an AI decision assistant, different ambiguity, missing data and adversarial inputs.

Strategic Analysis: Generalisation Is a Governance Question

Unseen scenarios reveal what training scores cannot

AAM-Gym's 678-aircraft evaluation scenario is particularly instructive because it was not used during training.

The algorithms had to generalise to unforeseen aircraft interactions.

The risk ratios remained below the unequipped baseline in the paper's experiment, but they worsened substantially compared with the 100-aircraft case.

That is exactly the kind of evidence leaders need before deployment.

The question is not merely:

Does the model work?

It is:

Under which operating conditions does performance degrade, and is the degradation acceptable?

Metrics must include system outcomes

AAM-Gym groups metrics such as safety measures and operational-suitability measures.

That is stronger than optimising one reward score.

An AI system may reduce one type of error while increasing another operational burden.

A scheduling algorithm may improve utilisation but create fragile queues.

A customer-service model may increase response speed but reduce resolution quality.

A maintenance model may reduce unplanned downtime but produce too many false alarms.

Governance therefore requires a metric portfolio.

Related article: Performance Systems Should Change Behaviour Before They Judge Results

Edge cases should be tested where the cost of reality is too high

The paper argues that simulation allows exploration of scenarios that may be too dangerous to test in the real world.

This is particularly important for safety-critical AI.

Real operations should not be the first place the organisation discovers how the system behaves under:

• extreme load;
• sensor failure;
• unusual interactions;
• conflicting objectives;
• or rare combinations of events.

Simulation creates a safer learning environment.

Reproducibility is part of governance

AAM-Gym uses structured configuration and records metrics for analysis.

The enterprise equivalent is versioned validation.

Leadership should be able to answer:

• which model version was tested;
• against which dataset or scenario;
• with which parameters;
• under which baseline;
• and what changed from the previous release?

Without traceability, validation becomes anecdotal.

Multiple simulators reduce model-of-the-world dependence

AAM-Gym was designed to be modular with respect to backend simulators.

That architectural decision has a broader governance value.

If a system passes only one simplified simulation environment, confidence may be overstated.

Where consequences are material, leaders should consider:

• multiple datasets;
• multiple simulation assumptions;
• independent evaluation;
• or real-world shadow testing.

No single test environment should become unquestioned truth.

Acceptance thresholds should be defined before the result is known

AI validation becomes weak when teams test a model first and decide afterwards whether the result is “good enough”.

That invites motivated reasoning.

Before evaluation, the organisation should define:

• minimum acceptable performance;
• maximum tolerable failure rates;
• scenarios that must be passed;
• conditions that require human fallback;
• and evidence that would block deployment.

The AAM-Gym paper notes that an agreed loss-of-well-clear standard for the AAM context had not yet been defined in that research setting. That is itself a useful governance reminder. A metric can be measured precisely even when the organisation has not yet agreed what acceptable performance means.

Measurement and acceptance are different tasks.

Human and organisational behaviour belong in the validation boundary

Even strong algorithmic performance may fail to create a safe or productive operating system if humans cannot understand, supervise or recover from the system's behaviour.

Enterprise validation should therefore test questions such as:

• Does the AI create too many alerts?
• Can operators recognise when confidence is low?
• Is there a practical handover path?
• Does the system encourage over-reliance?
• What happens when people disagree with its recommendation?
• Can the organisation operate safely when the AI is unavailable?

These questions are not peripheral to model performance.

They determine whether the model can become a dependable capability.

Decision Framework

Before allowing an AI system into consequential operations, require evidence across six layers.

The required depth should scale with consequence.

A low-risk drafting assistant does not need the same validation environment as collision avoidance, clinical decision support or autonomous industrial control.

From Strategy to Execution

Immediate action

For every material AI use case, define:

• operational objective;
• baseline;
• acceptance metrics;
• unacceptable outcomes;
• and deployment boundary.

Separate the training dataset from the validation environment.

Medium-term capability building

Create reusable simulation and evaluation infrastructure for recurring AI use cases.

Build scenario libraries containing:

• normal operations;
• high load;
• degraded inputs;
• unusual combinations;
• and known failure modes.

Automate regression testing so new model versions must demonstrate that performance has not deteriorated on critical scenarios.

Long-term strategic positioning

Treat validation infrastructure as an enterprise capability.

Organisations adopting AI at scale will eventually have many models, vendors and versions.

Without a standard evidence layer, every team will define “good enough” differently.

A shared validation architecture creates comparability, traceability and governance leverage.

It also enables faster adoption because leaders are not rebuilding the assurance process for every model.

Signals to Monitor

Leadership should become concerned when:

• the same data is used for training and final validation;
• a model is described as “better” without a baseline;
• only average performance is reported;
• edge cases are discussed but not tested;
• operational metrics deteriorate while model accuracy improves;
• model releases cannot reproduce prior test results;
• simulation assumptions are undocumented;
• or deployment scale materially exceeds the scale represented in validation.

Questions for the Leadership Team

1. What evidence would cause us to refuse deployment even if the headline model score is strong?
2. Which operating scenarios were not represented in training?
3. What is the real baseline the AI must outperform?
4. How does performance change as scale or complexity increases?
5. Which edge cases are too dangerous or costly to discover in live operation?
6. Can we reproduce the evidence for the exact model version currently deployed?
7. Who decides when simulation evidence is sufficient for operational trust?

Closing Perspective

AI governance cannot stop at policies and model cards.

Consequential AI systems need environments in which their behaviour can be challenged before the real world bears the cost of failure.

Simulation provides that opportunity when it is designed around representative scenarios, credible baselines, multiple metrics, unseen conditions and reproducible evidence.

The objective is not to prove that AI is perfect.

It is to know where it works, where it degrades, what happens when conditions change and whether the organisation has enough evidence to trust it for the decision at hand.