Making Better Project Termination Decisions
One indicator rarely tells you to stop. Several indicators pointing different ways is the normal condition of a development project, and it is the reason termination decisions arrive late.
Why one indicator at a time cannot support a kill decision
R5 is a short empirical article from 2002 in a practitioner journal for research and technology management. Its target is a specific habit: monitoring an ongoing R&D project by comparing performance against target values on indicators taken one at a time, and reaching for managerial experience when the readings disagree.
The study's own data make that concrete. The variable that discriminates most strongly at the start of development — whether product quality is held as a competitive priority — carried a canonical discriminant coefficient of 0.676 at the initial stage, the highest of any variable at that point. At the middle stage it was 0.45, still the highest. At the final stage it was 0.269, ranking seventh. All three are findings of this one study.
A monitoring regime that fixes its criteria at the first gate and applies them unchanged at every subsequent gate is therefore weighting a variable at the end that has lost most of its discriminating power. The consequence R5 identifies is not that projects get killed wrongly; it is that leaders seldom make termination decisions for ongoing projects in time, because attention was spent at selection and the ongoing signal is too noisy to force a decision.
How the variable set was built and reduced
The construction is worth following because it is reproducible on a smaller scale, and because the reduction step is where the twelve come from.
R5's method, as reported
Enumerate candidate variables
A literature review plus interviews with R&D managers and experts produced 41 candidate variables — a study parameter, not a recommended number.
Group them into six categories
The R&D project team · the market for the project output · the resources for developing the project · the technology · the priority of the project · the commercial goals.
Choose the measurement scale
Most variables were measured on a 1–7 Likert-type rating scale; a few were dummy variables taking only 0 or 1. Respondents were the project leaders or managers, answering on the development status of their projects.
Fix the evaluation points
Attention was restricted to the development stage, on the grounds that this is where the considerable resources are consumed. Within it, data were collected at three critical evaluation points: initial, middle and final.
Run stepwise discriminant analysis at each point separately
The dependent variable is project success versus failure. The procedure retains only the variables with the greatest joint predictive strength, so a variable that duplicates another's information drops out even if it correlates with the outcome on its own.
Read the coefficients
The output is a stage-specific set of discriminating variables plus a canonical discriminant coefficient per variable per stage — the measure of that variable's discriminating strength at that moment.
Twelve of the 41 variables survived as significant discriminators across the three stages. Not every one of the twelve enters at every stage: the final stage carries eleven coefficients, not twelve. Both counts are findings of this study.
The projects were undertaken between 1997 and 1999 across 17 industries, in manufacturing firms in one metropolitan area of one country. Classification into success and failure was made by the responding managers, which means the model is trained on their judgement, not on an independent outcome measure.
The twelve critical variables, ranked
R5 ranks the twelve in the order of their influence on the decision to terminate. The definitions below are close paraphrases of the study's own. Read the ranking as a finding about the 217 projects studied, not as a scoring weight to import.
THE TWELVE CRITICAL VARIABLES IN R5'S ORDER OF INFLUENCE
| Rank | Variable | Definition as given |
|---|---|---|
| 1 | Priority placed on product quality relative to competitors | Whether product quality is held as a competitive priority. Quality here denotes performance, security, appearance and utilisation cost. Has higher discriminating strength at all three stages. |
| 2 | Whether the project managers are technical experts or not | The technical ability of the managers, which to some degree determines project output. Technical ability prepares a manager to understand the development process and its prospects and to draw a proper development plan; it also confers prestige with which the manager can motivate team members. |
| 3 | Expected probability of technical success | Technical feasibility — how far the team comprehends and anticipates it, and judges technical capability. Explicitly dynamic: it rises as critical technical problems are resolved, and falls certainly and dramatically if emerging problems are not resolved in the expected time frame and new ones arise. |
| 4 | Work efficiency of the R&D manager | The managerial and operational ability of the project leader in terms of time consumption. Greater leader work efficiency, more likely to succeed. |
| 5 | Importance top management attaches to the project | The project's priority in the firm's portfolio. More top-management attention means easier acquisition of required resources and support from other relevant units, and therefore a higher chance of success. |
| 6 | Degree of freedom to do your work | Whether the environment is free and open enough for the team to develop and exert their capabilities. Justified on the grounds that R&D problems are often novel, with no prior experience to draw on, demanding creative thinking that a free internal atmosphere fosters. |
| 7 | Degree of urgency in project development | Whether the project admits no delay — its success is the prerequisite for other projects, or the market compels prompt launch. Explicitly two-sided: haste can degrade product quality, but urgency also attracts attention at all levels and stimulates the team. Net effect found positive. |
| 8 | Degree of transparency of critical decisions about the project | Whether the project team is informed in a timely fashion when R&D management or top management makes important decisions. |
| 9 | Degree to which chance events influence the project | Unexpected occurrences — the examples given are war and sudden raw-material price increases — that can influence the project positively or negatively. |
| 10 | Expected probability of commercial success | Probability of realising the expected market share. Tied to the stated main purposes of an R&D project: high market share and substantial return on investment. |
| 11 | Expected uses | The potential applications of the project output, which should be clearly defined and counted. Finding: the more uses identified during the development phase, the better the chance of success. |
| 12 | Efficiency of time consumption of managerial personnel | Whether managerial personnel work efficiently and make good use of their time. |
Complete list as ranked by R5. The order is that study's result on its own 217-project sample, 1997–1999, and is not a general weighting for R&D projects.
What the ranking says that most managers would not predict
Four features of this list run against the way termination criteria are usually written. Each is a finding of R5, and each is testable on your own portfolio.
Four surprises in the order
The top discriminator is a strategic posture, not a forecast
Rank 1 is whether product quality is held as a competitive priority against rivals. That is a statement about how the organisation has positioned the project, not a prediction about how it will turn out. The two forecast variables sit at rank 3 (technical success) and rank 10 (commercial success).
Expected commercial success ranks tenth of twelve
The variable most review boards reach for first is close to the bottom of the discriminating order in this dataset. It survived the reduction, so it carries information — but eight softer variables carried more.
Soft management variables outrank money and market
Manager technical expertise (2), manager work efficiency (4), top-management attention (5), freedom to work (6) and decision transparency (8) all outrank expected commercial success. Resources and budget do not appear among the surviving twelve at all, despite being one of the six original categories.
Urgency helped rather than hurt
R5 concedes the standard mechanism — haste degrades quality — but reports the net effect as positive, attributing it to attention and motivation. At the final stage the correlation between project output and the urgency variable was 0.345, described as a larger positive value. That is one study's finding, not a licence to compress schedules.
Reading three of the twelve as live signals
Most of the twelve are stable attributes you can score once a quarter. Three behave differently, and R5 treats them as quantities that move during the project — which makes them the ones worth instrumenting properly.
The three that carry a signal rather than a score
What this model can and cannot be used for
The limitations are specific and they bear directly on transferability. R5 states most of them itself, and one of its own recommendations is effectively an admission.
- Development stage only. The scope is termination during development, justified by resource consumption in that stage. Nothing is claimed about pre-development or post-launch termination.
- One metropolitan area, one country, 1997–1999, manufacturing firms across 17 industries. The paper frames its national context as one where the commercialisation ratio of R&D results is low and top leaders focus mainly on selection rather than ongoing monitoring.
- 135 of 375 questionnaires were valid — a 36 percent valid rate. The 217 projects are nested inside those 135 responses, so the projects are not independent observations.
- Single respondent type. Data are self-reported by project leaders or managers, retrospectively, with success or failure already known.
- Classification is the managers' own. A discriminant model can only discriminate between the classes it is given, and here those classes came from the responding managers.
- The authors recommend collecting data within one firm to increase the chance of a correct termination decision for that firm — which concedes that the cross-firm coefficients may not transfer to any single organisation.
It is also worth being clear about what kind of question this answers. R5 predicts success and failure as its respondents classified them; it does not tell you whether a project is worth its remaining cost. That is a valuation question, handled by the techniques in R&D project evaluation tools and risk and uncertainty in R&D financial analysis. A different empirical tradition again asks whether uncertainty is being resolved — see sources of uncertainty in R&D projects. Three questions, three models; a review that runs one of them and believes it has run all three will make confident mistakes.
What to carry forward
- The case against single-indicator tracking is not that indicators are wrong. It is that they conflict, and that their discriminating power changes as the project advances.
- Twelve variables survived reduction from 41. The complete ranked list is above; the top four are product-quality priority, manager technical expertise, expected technical success and manager work efficiency.
- Expected probability of commercial success ranks tenth, and no resource or budget variable survived at all. Both are findings of this study, and both are worth testing locally before you act on them.
- Three of the twelve move during a project — technical success probability, expected uses and product-quality priority. Instrument those; score the rest.
- Every number here belongs to one study of 217 projects in 17 industries, 1997–1999, self-classified by project managers. The authors themselves recommend re-estimating on your own firm's history.
Frequently asked questions
Can I use these twelve variables as a scoring sheet straight away?
As a checklist of what to look at, yes. As weights, no. The ranking is a result of one study's discriminant analysis on 217 projects in one metropolitan area between 1997 and 1999, and R5 itself recommends estimating the model on data collected within your own firm if you want a decision rule that is correct for that firm.
Why is expected probability of commercial success so far down the list?
R5 reports the rank without arguing it at length. One reading consistent with the method is that a forecast of commercial success carries information already present in other variables, so the stepwise procedure keeps the variables with greater joint discriminating power. What the finding does establish is that a review resting mainly on the commercial forecast is resting on a weak discriminator in this dataset.
What is a canonical discriminant coefficient, in plain terms?
It is the weight a variable carries in the function that best separates successful from failed projects. In R5 it is used as the measure of a variable's discriminating strength at a given stage — 0.676 for product-quality priority at the initial stage against 0.269 at the final stage, for example.
Does a low score on one of the twelve mean the project should be stopped?
No, and that is the whole argument of the paper. Success or failure depends on a combination of variables, and any single reading may conflict with others. The variables are inputs to a multivariate assessment, not individual kill triggers.
How many projects would we need to build our own version?
R5 does not state a minimum, and the library will not invent one. What the paper does tell you is the shape of the exercise: a classified history of completed projects, ratings collected at comparable points in development, and enough failures as well as successes for a discriminant procedure to have something to separate. R5's own base was 152 successful and 65 failed projects.
Is this the same thing as a stage gate?
No. A stage gate is a decision point with criteria attached; this is a model of which criteria carry discriminating power at a given point. R5's finding that the strongest early discriminator loses most of its power late is a direct challenge to gate criteria sets that apply identical weightings at every gate.
References and source attribution
- R5 — making better project termination decisions. Practitioner-facing empirical short article in a journal for research and technology management, January–February 2002; 3 printed pages; 3 references; no numbered tables or figures. Method: 41 candidate variables in six categories, measured on 1–7 rating scales, reduced by stepwise discriminant analysis run separately at three evaluation points in the development stage. Sample: 375 questionnaires distributed, 205 returned, 135 valid, covering 217 projects (152 successful, 65 failed) undertaken 1997–1999 across 17 industries in manufacturing firms in one metropolitan area.
- The prior work R5 draws on for the claim that success and failure factors vary drastically at different evaluation points is cited within R5 and was not supplied to this library.
- Eleven copyrighted journal articles on R&D project management, supplied as a reading set for a literature review and profiled for this library. Front matter, abstracts, framework sections, tables and figures were read; article bodies were not reproduced, and all content here is paraphrase. The set is a reading list, not a systematic or representative survey of the field.
- Supplied teaching source for this library (research methods and research process materials). Used here for page conventions and voice only; it does not treat project termination.
Suggested questions for Ask KEVOS
- Turn the twelve critical variables into a monitoring sheet for a project currently in development.
- Which of our current review criteria correspond to variables that survived R5's reduction, and which do not?
- Explain why a variable can correlate with success and still be dropped by a stepwise discriminant procedure.
- What data would we need to estimate a stage-specific termination model on our own project history?
- Draft a short paper arguing against single-indicator tracking for our R&D governance forum.
- How should we record the expected uses of a project output so the count is comparable across projects?