Expert Judgement and peer review

Every Performance Solution reaches a point where the numbers stop. Some questions have no Verification Method, no test data and no calculation that would be proportionate to the problem. For those, the NCC accepts Expert Judgement as an Assessment Method. It is the most flexible method and the easiest to abuse, because a confident opinion and a sound judgement can look alike on the page.

This chapter sets out what makes a judgement sound, writes the running case’s four judgements in that form, and follows a peer review that undermines the most important of them. The review’s finding is what finally decides the owner’s choice in Chapter 10.

What the NCC means by Expert Judgement

NCC 2022 Volume One defines Expert Judgement as the judgement of an expert who has the qualifications and experience to determine whether a Performance Solution or Deemed-to-Satisfy Solution complies with the Performance Requirements. Two parts of that definition carry weight. The person must be an expert with relevant qualifications and experience, and the judgement must be about compliance with the Performance Requirements, not about whether a design is sensible or popular.

The idea of recording the basis of a judgement is not new. The supplied BCA 2004’s introduction recommended that, where an Alternative Solution was proposed, the documentation should include details of any Expert Judgement relied on, including the extent to which it was relied on and the qualifications and experience of the expert. That is still the right standard. A judgement whose author, basis and extent are unrecorded cannot be reviewed, and a judgement that cannot be reviewed should not carry a building’s fire safety.

The anatomy of a sound judgement

A sound judgement has six parts. Leaving any of them out weakens it.

Figure 1The parts of a written Expert Judgement

The claim should be narrow. “The proposal is safe” is not a claim a judgement can support; it is the conclusion of the whole solution. “A fire starting within the 3 m clear zone at the proposal’s stair foot is less likely than a fire starting near the reference stair foot in the fabrication area” is a claim that a judgement can address.

The running case’s judgements

This is an illustration. The final report relies on four Expert Judgements:

1. Stair foot exposure (scenarios B and C). A fire threatening the proposal’s stair foot, in a 3 m clear zone beside exit P2, is less likely than a fire threatening the reference stair foot in the fabrication area, where hot work and combustible packaging are routine.
2. Wayfinding (E4P2). With direction signs at each aisle junction on the mezzanine, an unfamiliar occupant can find the stair in conditions of reduced visibility as readily as in the reference layout, whose route has fewer turns.
3. Shelving as an obstruction. Floor-fixed shelving is treated as a built obstruction for measuring travel distance, so routes are measured along the aisles.
4. Scenario D credibility. With the mezzanine restricted to non-combustible storage, a fire on the mezzanine large enough to threaten its occupants before they can leave is not a credible design scenario, so the 7.65 s difference in scenario D does not represent a reduction in safety.

The first three are modest and well bounded, and each could be checked by walking the building. The fourth carries the solution: without it, scenario D fails and the brief’s criteria are not met.

Peer review

The brief required an independent peer review by a second fire safety engineer, not involved in the design, before the final report was issued. The reviewer’s scope was the brief, the inputs, the calculations, the judgements and the conditions. Peer review is not required by the NCC for every Performance Solution, but it is a sensible measure for any solution that rests heavily on one judgement, and the certifier welcomed it.

The reviewer’s findings on the calculations have already appeared: the straight-line measurement in Chapter 3 and the vertical stair distance in Chapter 6. Both were corrected. The findings on the judgements were more consequential.

Figure 2Peer review findings on the four judgements

The finding that changed the outcome

The reviewer read judgement 4 against the mezzanine drawings and the stores procedure, and found two problems.

First, the mezzanine deck was specified as structural particleboard flooring on steel joists, which is common for mezzanine kits. A rule that only non-combustible items may be stored on the mezzanine does not make the mezzanine itself non-combustible. A fire starting on the deck, for example from an electrical fault in a light fitting or a dropped hot part from the machine shop, would have fuel underfoot.

Second, the stores procedure listed “tooling in steel bins”, but the tooling arrived in moulded plastic cases, and many cutting tools were kept in their plastic packaging to protect their edges. The judgement assumed a storage pattern that the business did not actually follow.

The reviewer’s conclusion was that judgement 4 claimed too much. A fire on the mezzanine would be less likely and smaller under the storage rule, but it was not shown to be “not credible”. The reviewer suggested that the judgement could be supported only if the deck were changed to a non-combustible floor such as steel plate, and plastic cases and packaging were banned from the mezzanine, which would mean unpacking every tool into steel storage.

The error: overstating a judgement

The original wording of judgement 4 was itself the error, and it is a common one. The engineer had a reasonable intuition, that restricting storage greatly reduces the risk, and expressed it as a categorical statement that the scenario was not credible. Categorical language claims more than the basis supports, and it hides the conditions on which the judgement depends.

The detection was straightforward: the reviewer asked, for each judgement, “what would have to be true for this to be wrong?” For judgement 4, the answer was “anything combustible on the mezzanine”, and the drawings and procedures showed two such things. Asking that question of one’s own judgements before a reviewer does is the single most useful habit in this chapter.

The revised wording the reviewer would accept read: “A significant fire on the mezzanine is less likely under the storage restriction than without it, but is not eliminated while the deck is combustible and tools are stored in plastic cases. Scenario D remains a credible scenario in which the proposal’s escape time exceeds the reference’s by 7.65 s.” That is honest. It also means the solution does not meet the brief’s criteria without further design changes.

The engineer did not resist the finding. A judgement is a professional opinion, and changing it in the face of a better-informed view is part of the job, not a loss of face. What would have been a failure is defending the original wording because the report was nearly finished, or quietly dropping scenario D from the report so that the rejected judgement was no longer needed. Both happen, and both are what peer review exists to prevent.

Judgement by analogy

Many judgements rest on analogy: “this is like a building I have seen perform well”, or “this arrangement is common and has not caused problems”. Analogy is a legitimate basis, and experienced engineers rely on it constantly. It is also where judgements most often fail, because the analogy holds in the respects the expert noticed and breaks in a respect they did not.

The test is to list the respects in which the analogy is claimed to hold and then look for differences that matter to the claim. Judgement 4 in the running case drew on an analogy with steel-stock mezzanines in fabrication shops, which hold bar, plate and fittings with almost nothing that burns. The analogy held for the stored items the engineer had in mind. It failed for the deck, which in those mezzanines is usually steel plate, and for the plastic tool cases, which steel stock does not have. A list of “respects in which this mezzanine resembles the analogue” would have shown both differences.

What a peer reviewer should and should not do

A peer review is valuable only if its scope and independence are clear. In the running case, the brief set four rules for the reviewer:

• Review, do not redesign. The reviewer tests whether the brief, inputs, calculations and judgements support the conclusions. Suggestions for alternative designs are welcome but are not the review’s purpose.
• Check the inputs against the brief. Every input used should be the one agreed, and any departure should be explained.
• Recalculate independently. At least the critical results should be reproduced from the inputs, not just read. The vertical stair error in Chapter 6 was found this way.
• Write the findings down. Each finding should state what was checked, what was found and whether it changes a conclusion, so that the engineer can respond point by point and the certifier can see the exchange.

The reviewer should also be independent in fact: not employed by the same firm on the same project, not the designer, and not paid on the basis of the outcome. A review that the author’s colleague completes in an afternoon may catch arithmetic errors but is unlikely to challenge a judgement the colleague shares.

Who is an expert?

The definition asks for qualifications and experience relevant to the judgement. For fire safety judgements in a Performance Solution, that usually means a fire safety engineer with experience of the building type. Several states register engineers, and some require registration for particular kinds of engineering work, including in some cases fire safety engineering. The engineering registration, competence and duty of care chapter covers the wider principles.

Expertise is specific. The owner’s operations manager is an expert on how the stores work and was right to point out the export crates in Chapter 2. That makes them an essential source of facts for the judgement, not the person who should make it. The judgement about whether a fire on the mezzanine is credible belongs to someone with fire engineering qualifications and experience, using the operations manager’s facts as part of the basis.

Recording reliance

The final report must say how much each conclusion depends on each judgement. In the running case:

• Scenario A’s conclusion rests on calculation and on the detection evidence, with judgement 3 setting the measurement basis. Reliance on judgement is low.
• Scenarios B and C rest almost entirely on judgement 1. Reliance is high, but the claim is narrow and supported.
• E4P2 rests on judgement 2 and on sign evidence. Reliance is moderate.
• Scenario D would have rested entirely on judgement 4, which the reviewer did not accept.

Writing this down shows the certifier, and the owner, exactly where the solution is strong and where it depends on opinion. It is the modern form of the BCA 2004 advice to record “the extent to which the judgement was relied upon”.

Figure 3Expert Judgement and peer review record

Exercise: write a judgement properly

Rewrite judgement 2, on wayfinding, in the six-part form, using the facts in this chapter.

Worked answer. One acceptable version:

• Claim: With the specified direction signs, an occupant unfamiliar with the mezzanine can locate the stair from any point on it, in conditions of reduced visibility, as readily as in the DTS reference layout. This addresses E4P2.
• Expert: The project’s fire safety engineer, with stated qualifications and experience of industrial buildings.
• Basis: The mezzanine and shelving plan; the proposed route of 28 m with two right-angle turns; the reference route of 20 m with one turn; the sign positions and the evidence for the sign type; the emergency lighting design for the aisles.
• Reasoning: Each decision point on the proposed route, meaning each aisle junction, has a direction sign visible from the approach, so an occupant is never required to choose a direction without guidance. The reference route has fewer decision points but no more guidance at each. The additional turn is therefore guided rather than left to memory.
• Limits: The judgement assumes the shelving layout shown; any change to aisles invalidates it. It does not address conditions of very dense smoke, which are covered by the escape-time comparison.
• Conditions: Shelving layout fixed as drawn; signs maintained and visible; emergency lighting maintained; any change to the aisles reviewed by the engineer.

A judgement in this form can be checked, challenged and maintained. A sentence such as “the route is easy to follow” cannot. The six-part version is longer, but every extra line is something a reviewer, certifier or future owner would otherwise have to guess.