Barriers and Maturity in Post-Project Review
Only one of the four obstacles R8 identifies is procedural. The other three are cognitive, social and epistemic — which is why organisations that fix the process still find the reviews hollow.
Why the barriers are not mainly procedural
R8 is a 2003 practitioner-journal study of post-project reviews in R&D, combining a survey of 63 R&D managers with an interview programme. Its diagnostic contribution is a four-cluster map of what stops organisations learning from their own projects, and the map's shape is the argument: three clusters have nothing to do with how the review is run.
The four principal barriers
Psychological
Inability to reflect, and memory bias. What people remember is selected before anyone asks them a question.
Team-based
Reluctance to blame, and poor internal communication. Looking back at problems puts collegial relationships at risk and softens the useful feedback.
Knowledge-utilisation
Difficulty generalising, and the tacitness of process knowledge. The part that would transfer is the part hardest to put into words.
Managerial
Time constraints, and bureaucratic overhead. The only cluster a process improvement reaches, and the one organisations address first.
This page is the diagnostic half of R8. How to run a review — the definition, the six conduct rules, the timing choice, the attendee list and the output — is on post-project reviews in R&D.
Psychological and team-based barriers
Inability to reflect, and memory bias
R8 starts from a general claim: all human beings are limited in their capacity to learn from experience, and natural evolution has programmed a balance between reflection and action. It offers two mechanisms without choosing between them — an inability to evaluate the link between results and possible initial conditions, or a sense that accelerating technological change makes reflection a losing battle. Either way the result is an indisposition to reflect on past actions and their consequences.
Beyond selectivity of reflection sits selectivity of memory. R8 treats repression as a natural way of limiting everyday complexity, well described at the individual level in psychology, and then names a gap: team-based repression of information is still poorly understood.
Reluctance to blame, and poor internal communication
Reviews look back at problems and critical phases, so friendships and collegial relationships are at risk when co-workers point a finger at one another. R8's consequence is more damaging than silence: for the sake of smooth cooperation in future projects, personal feedback is softened and often rendered ineffective.
R8 adds a composition effect. Teams made up of people in different locations, with different technical, functional or cultural backgrounds, or with individual rather than team objectives, are less likely to share critical information before and during the review — and often hold irreconcilable interpretations of common experiences. The barrier is partly a consequence of how the team was assembled, months before anyone scheduled a review.
Knowledge-utilisation and managerial barriers
Difficult to generalise, and the tacitness of process knowledge
In R&D projects, technologies and know-how are applied to specific problems, so the resulting insights are context-specific. R8's position is strong: it is difficult, if not impossible, to generalise them across a wide range of projects — and that holds even where teams have the time and ability for systematic learning.
R8 attributes part of this to the tacitness of process-related knowledge. The classic observation that we know more than we can tell still applies: it is hard to articulate impressions into clear, succinct messages, and people will always struggle to convert tacit knowledge into explicit knowledge shareable across an organisation.
Time constraints, and bureaucratic overhead
R&D is under constant time pressure, leaving little opportunity to evaluate yesterday's work. R8's interviewed managers described a prevailing attitude that if you are thinking, you are not doing — time treated as the critical resource, retrospection as something to delegate.
The second half is compliance. Reviews create additional bureaucratic overhead, and R8 reports that compliance with requests from project management departments is usually low. Two conditions make it lower: where the request is incompatible with the team's own informal project checkpoints, and where the team has little confidence in the management control system.
THE FOUR BARRIERS AND WHERE EACH ONE BITES
| Cluster | The two named mechanisms | When it operates | What it defeats |
|---|---|---|---|
| Psychological | Inability to reflect; memory bias | Before the review, as experience is encoded or dropped | Any technique assuming participants can recall what happened |
| Team-based | Reluctance to blame; poor internal communication | During the review and its preparation | Honest content, including on a technical-only agenda |
| Knowledge-utilisation | Difficult to generalise; tacitness of process knowledge | After the review, as insight becomes a transferable statement | Transfer to any project that is not closely similar |
| Managerial | Time constraints; bureaucratic overhead | Around the review — scheduling, resourcing, compliance | The review happening at all, and at usable length |
Clusters and mechanisms from R8's four-lobed barrier diagram. The right-hand columns are this library's synthesis; the source does not put the barriers on a timeline.
The five-level maturity model
R8's second framework treats review capability as an organisational competence with identifiable levels, explicitly modelled on a software-development capability maturity model. The paper offers it as a convenient metaphor for thinking about your practices and identifying the right steps to advance — a diagnostic, not a certification scheme.
R8'S FIVE-LEVEL POST-PROJECT REVIEW MATURITY MODEL
| Level | Label | Defining characteristics | What must be done to reach it |
|---|---|---|---|
| 1. Initial | Ad hoc reviews without guidelines | Reviews ad hoc or even chaotic; where a process exists, success depends largely on the skills and talent of the individuals conducting it. Seldom planned — triggered in reaction to a major project event, a complete failure or a conspicuous success. Process poorly defined, execution not standardised, outcome unpredictable and not comparable between reviews | Baseline. R8 reports most organisations still appear to be here |
| 2. Repeatable | Guidelines for comparable reviews | Managing and planning new reviews draws on experience with similar earlier ones, though specific practices may still differ. With principal review policies in place, major disasters in review sessions or outcomes may be avoided. Practices likely still focused on technical and financial data, but a first step toward transferable recommendations for corrective action | Establish review guidelines and sound practices — for example, post-completion reports on pre-defined templates |
| 3. Defined | Standardised reviews and identified output | Reviews documented, standardised and integrated into the overall project management process, for both management and engineering, and conducted according to consistent company-wide criteria. Major savings in cost and time become possible, as do quality gains | Put the system at the core. Train teams to collect information during the project — logbooks, and records of events useful for future corrective action. Assign training and supervision of review practice to a small project management department |
| 4. Managed | Actionable failure tolerance | The organisation accepts that failure is a necessary part of innovation and is prepared to act on it. It sets quantitative quality goals measurable across review outcomes, and results are collected and made available company-wide. The quality of the transferable knowledge is predictable and actionable, so full quality-management practice can be applied to reviews themselves | More than new protocols. Introduce incentive systems for innovation — R8's example is an award for teams whose projects failed but who learned and passed on the experience, with the caveat that you should not receive too many |
| 5. Optimizing | Reviews are reviewed | The review practice itself is regularly reviewed, so flaws in the process are detected early and dealt with proactively. Reviews are established organisation-wide for consistent inter-project learning, and lessons are disseminated to targeted project teams elsewhere | Precondition: an organisational culture devoted to learning. R8 offers no shorter route |
Levels, characteristics and advancement steps from R8's five-step staircase figure and accompanying text. The source presents the model as a benchmark for self-location; it is normative rather than empirically derived.
Two steps map onto the barriers. Two to three is about when information is captured — collecting during the project rather than reconstructing afterwards attacks memory bias. Three to four is about what the organisation will say out loud about failure, which attacks the team-based barrier.
Where organisations actually sat
R8 distributes its survey percentages across the level descriptions rather than collecting them, so they are consolidated here. Every figure is a finding of one survey of 63 R&D managers, collected in 2000 and 2001 — none is a rate for the field.
SURVEY MARKERS BY LEVEL — STUDY FINDINGS, 63 RESPONDENTS
| Level indicated | Marker | Proportion reporting it |
|---|---|---|
| Below level 1 | Never conducts post-project reviews | One in eight |
| Level 1 | No formal guidelines on how reviews are conducted | 55.6% |
| Level 1 | Review quality based mostly on team members' capabilities | About a quarter |
| Level 1 | Of those that review, projects selected arbitrarily or at random | Half |
| Level 2 | Reviews follow practices similar to earlier projects | 17.5% |
| Level 2 | Sound review practices all employees are familiar with | 11% |
| Level 3 | Company has defined review practices | 11.1% |
| Level 3 | Sound, consistent criteria on every review | 6.3% |
| Level 3 | Dedicated project management unit, guidelines on request, or substantial review time on project methods | Fewer than 20% |
| Level 4 | Reviews have quantified and measurable goals | About 10% |
| Level 4 | Review quality independent of project and people involved | About 10% |
| Level 4 | Results used to improve future project management | 12.7% |
| Level 5 | Review processes regularly assessed and improved | 9.5% |
| Level 5 | Reviews an integral part of inter-project learning | 3.2% |
Findings of R8's survey, consolidated from statements distributed across the paper; level attribution follows the paper's own placement. Respondents attended two executive training events on innovation management — a self-selected population already interested in the topic.
R8's corroboration comes from elsewhere: a benchmarking study of 79 highly regarded R&D organisations found fewer than a quarter made full use of post-project reviews. That is an externally cited figure quoted by R8, not a result of its survey; R8 is not itself evidence for it.
The level-two trap, and how to assess yourself honestly
R8 names a specific failure of ambition. Many organisations stop at Repeatable because they believe they obtain the greatest learning for the effort invested there — templates exist, reports get filed, the ratio looks efficient.
Locating your own practice
Conditions R8 says are outside the reviewer's control
- The size of the project and the team
- The significance of the project for the organisation's success and image
- The project management style in use, and team leadership
- R8's point: practices differ for project-related, industry-related and cultural reasons, and there is no single right practice
R8 closes on that note rather than on the model. Whether or not you use its five levels, what matters is the practice of applying reviews and the willingness to improve what you are doing — and the paper is candid that 63 responses is too small for in-depth statistical evaluation. It also reached this library inside a reading set assembled for a literature review, not a systematic survey: see eleven R&D management papers compared. Where the barriers point at how decisions are owned rather than at the review itself, see governing R&D decisions organisationally.
What to carry forward
- Three of the four barrier clusters are not procedural. Improving the review process addresses one, and it is the one most organisations start with.
- Restricting a review to technical matters does not neutralise blame dynamics — feedback is softened even in a purely technical context.
- A poorly resourced review can be worse than none, because complex observations get extrapolated into simplistic general rules the organisation then acts on.
- The maturity model runs from ad hoc reviews to a practice that reviews itself. Level three moves information capture into the project; level four makes failure sayable.
- Level two is where practices stall, because the learning-per-effort ratio looks best there and the output is documents rather than obligations.
- Assess maturity against artifacts, not impressions. R8 found self-assessed maturity inflated and said so about its own figures.
Frequently asked questions
If the barriers are mostly not procedural, is there any point improving the review process?
Yes, but expect it to address one cluster of four. Better scheduling, templates and facilitation attack the managerial barrier and part of the team-based one. Memory bias is attacked by capturing information during the project, and the generalisation problem is attacked by being disciplined about what an insight actually supports.
What is the fastest way to move from level one to level two?
R8's answer is guidelines plus sound review practices, with post-completion reports based on a pre-defined template as the worked example. That produces reviews comparable enough to avoid the worst outcomes. It is a real step, but R8's warning is that most organisations then stop there.
Why does the model treat tolerance of failure as a maturity level rather than a culture question?
Because at level four it has to be operational. The organisation accepts failure as a necessary part of innovation and is prepared to act on it, sets quantitative quality goals for reviews, and circulates results. R8's illustrative mechanism is an award for teams whose projects failed but who learned and passed on the experience — with the caveat that you should not receive too many.
Can we use these survey percentages as a benchmark?
No. They are findings of one survey of 63 R&D managers recruited at two executive training events, which is a self-selected population already interested in the topic and would if anything bias the picture optimistic. R8 states the sample is too small for in-depth statistical evaluation.
How do we tell whether we are really at the level we think we are?
Ask for the artifact. Guidelines document, criteria set, in-project logbooks, quantified review quality goals, evidence the review process itself has been examined. R8 found managers claiming the top level while counting one-off learning events, and concluded its own reported figures were probably optimistic.
Where does this page stop and the practical one start?
This page covers why organisations do not review and how to assess where a practice sits. The definition, the six conduct rules, the timing choice, who attends, what is asked and what the output must contain are on the companion page about conducting post-project reviews.
References and source attribution
- R8 — post-project reviews in R&D. Practitioner-facing mixed-methods study in a journal for research and technology management, September–October 2003; 7 printed pages; 13 references. Survey of 63 R&D managers representing more than 40 companies across more than 12 industries, collected at two executive training events in September 2000 and October 2001, supported by 27 interviews with R&D managers from 13 multinational companies conducted between 1997 and 2001. Sections used here: the four-barrier analysis and its diagram, the five-level maturity model and its staircase figure, and the survey results distributed across the level descriptions.
- The software-development capability maturity model on which R8 explicitly bases its five levels. Named by R8 as its structural source; not supplied to this library and not reproduced here.
- The benchmarking study of 79 highly regarded R&D organisations, fewer than a quarter of which were found to make full use of post-project reviews, and the earlier work on post-implementation evaluation of information systems. Both are cited within R8 as corroboration; R8 is not itself evidence for either.
- Eleven copyrighted journal articles on R&D project management, supplied as a reading set assembled by a student for a literature review and profiled for this library. Front matter, abstracts, framework sections, tables and figures were read; article bodies were not reproduced, and all content here is paraphrase. The set is a reading list, not a systematic or representative survey of the field.
- Supplied teaching source for this library (research methods and research process materials). Used here for page conventions and voice; it does not treat organisational learning or maturity models.
Suggested questions for Ask KEVOS
- Assess our post-project review practice against the five levels and tell me which artifacts we are missing.
- Which of the four barriers best explains why our reviews produce nothing actionable?
- Draft a plan to move our review practice from level two to level three in one financial year.
- What would we have to capture during a project to defeat the memory-bias barrier?
- Write the case for resourcing reviews properly, using the argument that a bad review is worse than none.
- How would we set quantitative quality goals for a review that can be compared across projects?