A project has been running for eleven months. The monthly report has been green for nine of them and amber for two. Spending is within limits, milestones are mostly met, and the people reviewing it have asked sensible questions. Then, within a couple of months, the picture changes completely. Nothing dramatic happened. A set of conditions that had been drifting for most of a year finally became impossible to present as anything other than what they were.
The usual diagnosis is that someone was managing the message. Sometimes that is true. More often the reports were accurate every month. The problem was that they measured things that could not move until the situation had already deteriorated. The most important property of any reporting system is not what it measures, but how long it would take to tell you that you were wrong.
This article explains detection speed, why more measures rarely improve it, how to build monitoring into a decision at the moment it is made, and how to decide how closely to watch work you have handed to a supplier, including whether you have someone able to understand what they see. It is general information for owners and managers who rely on reports, projects and suppliers.
Detection speed is not reporting frequency
Detection latency is the time between something changing and the business becoming aware of it. It is different from how often you look. A weekly report can have a four-month latency if everything on it is a lagging measure. A quarterly review can have a two-week latency if the right things are measured continuously and problems can be raised between reviews.
Four common misreadings:
- More measures reduce latency. Usually they do not. Sixty lagging indicators detect change no faster than six, and real movement is harder to see among them.
- Status colours are information. A red, amber or green rating is someone’s judgement, made by a person with incentives. When the underlying data has not moved but the colour changes, what changed is someone’s willingness to say so.
- No bad news means a good position. A quiet report may just mean the measures are insensitive to what matters.
- Measuring the plan measures the outcome. Being on schedule and on budget shows conformance to assumptions. If the assumptions were wrong, a project can conform perfectly to an irrelevant plan.
Measures that move early
Think of a reporting system as an instrument with a response time. Engineers would not install a temperature sensor with a four-hour lag on a process that can fail in twenty minutes, then conclude from a stable reading that all is well. The same logic applies to business reports, but the question “how long would this take to tell us we are wrong?” is rarely asked when reports are designed.
Measures with genuinely short latency tend to share three properties:
- They can move before money does. Anything that only changes after cost is incurred lags by design.
- They measure behaviour, not just delivery. Delivery measures show whether you did what you planned. Behaviour measures show whether it is working: whether customers use the new service, whether staff follow the new process, whether workarounds are appearing.
- They are cheap to collect continuously. A measure that needs a special exercise is collected only when someone is already worried, which is when its early-warning value is gone.
For each significant initiative, ask:
| Test | Question | Good answer |
|---|---|---|
| Latency | If our main assumption became false today, how long until we knew? | Weeks, stated as a number |
| Precedence | Can any measure move before spending does? | At least two can |
| Behaviour | Do we measure what people do, not just what we delivered? | Yes, separately |
| Falsifiability | What observation would show we are wrong? | Agreed in advance |
| Ownership | Who must raise a deviation, and to whom? | Named, with a route outside the monthly cycle |
| Cost of looking | Does the measure need a special exercise? | No, it is always available |
If nobody can give a latency figure at all, that is the finding.
It also helps to separate two kinds of report. A detection report exists to show early movement, and it should be short and allowed to look alarming. An assurance report shows that things were done properly. Combining them tends to push early indicators out, because they look worrying, and fill the report with material that demonstrates diligence. The reports that change decisions article covers putting forecasts and decisions at the front of a report.
Build monitoring into the decision
When a business accepts a risk or makes an uncertain decision, the decision is only complete when it says how the business will learn whether it was right. Treat monitoring as part of the decision, agreed at the same time:
- the assumption being made;
- the indicator that will test it;
- how often evidence is needed;
- who reviews it;
- the threshold that triggers action;
- the action, and whether it is pre-authorised;
- when monitoring can be reduced or stopped.
Monitor the control, not just the outcome. If the risk is a fuel or chemical leak, waiting for contamination is too late; inspection results, bund condition and alarm tests move earlier. Monitor expected benefits as well as possible harm: if a project was justified partly by savings or improvements, test them too. And look for side effects: a change that reduces one problem can create another elsewhere.
Finally, make sure what is learned survives the project. Monitoring results often stay in a closed project folder, and the next job repeats the same assumptions. Update templates, standards and estimating data when the evidence justifies it. The plans that detect rather than predict article covers designing plans around the assumptions that matter most.
Make it easy to raise a concern between reports
The fastest indicator in any business is a person who notices something is wrong. That signal is often lost because the only formal route is the next monthly report, by which time the observation has been softened, summarised or forgotten. Set up a route that bypasses the reporting cycle: a named person who can be told directly, an agreed way to flag an issue the same day, and a clear expectation that early warnings are welcome even when they turn out to be false alarms. A false alarm costs a conversation. A missed warning can cost months.
How far can you see into work you have outsourced?
When work is handed to a supplier, the business usually sees whatever the supplier chooses to report, at the frequency and in the format set during contracting. That is rarely decided deliberately; it is inherited from a template.
Research by Graham Winch and colleagues, published in 1998, borrowed a term for this: the line of visibility, meaning how transparent the delivery process is to the client. They argued it should be an explicit early choice. In their case study of a large research facility, the client placed many of its own people on the project, attended all significant design reviews and set up a small internal team of engineers experienced in the relevant plant to review the designers’ work. The authors reported that the changes this team proposed were costed at a substantial saving and, more importantly, that because the client took part in every significant review, it could not be surprised.
Visibility has two separate settings:
- Depth: which meetings, tests and decisions you attend, and which documents you see in working form rather than as a summary.
- Capability: whether the people who attend can understand what they see, and have the standing to say when something is wrong.
Access without capability buys observation, not oversight. And visibility is not free: it costs your people’s time. The choice is between two coherent designs:
- Presence buys early knowledge and the chance to influence decisions while they are cheap to change, at the cost of your time and some blurring of accountability.
- Exception reporting is cheaper and keeps accountability clean, but you learn of problems only when a limit is crossed, which is after the fact.
In practice, the main return from visibility is not catching dishonesty, which is rare. It is discovering mismatched assumptions: that you and the supplier mean different things by “complete”, “tested” or “ready”, while each reports accurately against its own understanding.
Two tests help:
- The empty-chair test. For each meeting or test you have the right to attend, who would go, and could their substitute understand the material? If the seat would be empty, the visibility is only on paper.
- The surprise review. After any unwelcome surprise, ask where you could have learned of it earlier, and whether you were there. “We were not invited” is a depth problem. “We were there and did not notice” is a capability problem.
A worked example
This is an illustration. A regional food producer is making two changes at once: moving a new product line to a contract packer, and launching online ordering for its wholesale customers.
Online ordering. The business case assumes that within three months, about 70% of wholesale orders will come through the new system, freeing roughly 15 hours a week of phone order-taking. The original plan was to check this in the quarterly sales review, a latency of up to three months. The owner instead sets up a weekly measure from the ordering system: the share of orders placed online, and the number of customers who logged in once and never again. The monitoring is written into the decision: if online orders are below 40% after six weeks, the sales coordinator calls the non-adopters and the developer reviews the ordering screens; monitoring drops to monthly after six months if the share is stable.
By week four, online orders are 22%, and twelve customers logged in once and stopped. Calls reveal the cause within a day: the system makes customers re-enter their usual order every week. A “repeat last order” button is added. By week ten, online orders are 64% and rising.
The contract packer. The supply agreement gives the producer the right to attend production runs and see batch records, but nobody had been assigned to do it. Under the empty-chair test, the seat was empty. The owner assigns the quality coordinator, who understands the product’s specifications, to attend the first three runs and visit quarterly. At the first run, the coordinator notices the packer is measuring fill weight differently from the producer’s specification. Both were reporting accurately against their own understanding. The difference is fixed before any product ships.
How this applies to a small Australian business
- Ask what you would see first, and how long after, for each significant initiative.
- Add behaviour measures to delivery measures.
- Choose measures that move before money does.
- Separate detection reports from assurance reports.
- Write monitoring into each significant decision: assumption, indicator, threshold, action and end date.
- Monitor controls and benefits, not just outcomes and harm.
- Decide deliberately how far you will see into outsourced work, and assign someone capable to look.
- Review surprises to learn whether the problem was depth or capability.
Signals worth watching
- Reports that stay green until a sudden collapse.
- Status colours changing while the underlying data has not.
- Measures that only move after money is spent.
- Rights to inspect supplier work that nobody uses.
- Monitoring data that disappears when the project closes.
- Surprises that someone, somewhere, could have seen earlier.
Common mistakes
- Adding measures instead of shortening latency.
- Treating status colours as evidence.
- Measuring delivery and ignoring behaviour.
- Treating monitoring as an afterthought rather than part of the decision.
- Buying access without the capability to use it.
- Assuming a quiet report means a sound position.
Frequently asked questions
How do we estimate detection latency? Pick your main assumption and ask what would change first if it were false, where that would show, and how often you look there. The total is your latency.
Do we need expensive data systems? Rarely. Many early measures come from systems you already have: order data, rosters, complaint logs, job records.
How much should we watch a supplier? In proportion to how critical and how uncertain the work is. Attend early runs, key tests and major decisions; rely on exception reporting for routine work.
What if we lack the expertise to review supplier work? Decide whether to build it, borrow it from an adviser for key moments, or accept exception reporting knowingly.
When can monitoring stop? When the assumption it tests has been confirmed for long enough, or the risk has changed. Decide the end point when you set it up.
Questions to ask
- If our main assumption were wrong today, how long until we knew?
- Which of our measures can move before money is spent?
- Do we measure what people do, or only what we delivered?
- What monitoring did we agree when we made our last big decision?
- Which supplier meetings or tests do we have the right to attend, and who goes?
- Where did our last surprise come from, and could we have seen it sooner?
Bringing it together
The most important property of a reporting system is how quickly it would tell you that you were wrong. Shorten that time with a few measures that move early, especially measures of behaviour, rather than with more lagging indicators. Treat monitoring as part of every significant decision, with an assumption, indicator, threshold, action and end point agreed at the start. Decide deliberately how closely to watch work you have outsourced, and make sure someone capable is in the room. Early knowledge is cheap; late knowledge is expensive.
Source: KEVOS notes, drawing on J. Portny’s guidance on monitoring work under uncertainty, environmental impact assessment and risk guidance on monitoring and audit, and G. Winch, A. Usmani and A. Edkins (1998), Construction Management and Economics, on the client’s line of visibility. Examples and figures in this article are illustrations. This article is general information.
