Reading research critically: judging studies, summaries and claims before acting on them (Research and Evidence for Better Decisions) | READOS

Business decisions are increasingly justified by research. A vendor cites a study showing that its software cuts downtime by 40%. A consultant’s report concludes that four-day weeks raise productivity. An industry article claims that a particular coating doubles service life. A manager forwards a news story about a survey showing that most customers prefer a certain feature. Each claim may be sound, partly true or misleading, and the difference often lies in details that headlines and summaries leave out: how many cases were studied, how they were chosen, what they were compared with, how large the effect really was and who paid for the work.

Reading research critically does not require advanced statistics. It requires a set of questions asked consistently: what exactly was studied, how, by whom, compared with what, with what result and how far that result can be applied to the situation at hand. These questions protect against both credulity, accepting claims because they sound scientific, and cynicism, dismissing all evidence because some is weak. They also help when the task is the reverse: summarising a body of research for colleagues who need to decide.

This article explains how studies are structured, the questions that reveal their strength, common biases and traps, the difference between statistical and practical significance, how to judge whether results apply to a particular business, how to review a body of literature and how to summarise evidence for decision-makers. It is part of a series on research and evidence for business decisions.

The anatomy of a study

Most research reports, whether academic papers or industry studies, contain the same elements, even if they are not labelled:

the question the study set out to answer;
the method: what was done, to whom or what, and how outcomes were measured;
the sample: who or what was studied, how many and how they were chosen;
the results: what was found, ideally with numbers and measures of uncertainty;
the conclusions: what the authors believe the results mean; and
limitations: what the study cannot show.

A useful reading habit is to read the method and results before the conclusions. Conclusions are the authors’ interpretation, and they sometimes go further than the results justify.

Who did it, and why

Ask who conducted and funded the research, and what they stood to gain. Research funded by a vendor, an industry group or an advocacy organisation is not automatically wrong, but it deserves closer scrutiny, particularly when the methods and data are not available. Independent replication, publication in a reviewed outlet and transparency about methods and data all increase confidence.

The sample

The sample determines how far results can be trusted and generalised:

Size: very small samples produce unstable results that can easily occur by chance.
Selection: how were participants or sites chosen? Volunteers, enthusiastic early adopters and the vendor’s best customers are not typical.
Response: in surveys, what share responded, and might responders differ from non-responders?
Relevance: do the people, businesses or conditions resemble your own?

The comparison

A result means little without a comparison. A claim that downtime fell by 40% after installing a system raises immediate questions: compared with what period, and what else changed at the same time? Stronger designs compare the treated group with a control group that did not receive the change, ideally chosen at random, or compare trends before and after against similar sites that did not change. Without a credible comparison, improvements may reflect seasonal patterns, other initiatives, regression to the mean after an unusually bad period or simply chance. The what would have happened anyway article explores this question in business decisions.

Correlation and causation

Many studies report associations: businesses that do X tend to have higher Y. Associations do not prove that X causes Y. A third factor may drive both, the direction may be reversed, or the association may be coincidental. Causal claims need designs that rule out these explanations, such as randomised experiments, natural experiments or careful comparisons that account for confounding factors.

The size of the effect

A result can be statistically significant, meaning unlikely to be due to chance alone, without being practically significant, meaning large enough to matter. With large samples, tiny effects become statistically significant. Look for the size of the effect in meaningful units, such as hours saved, defects avoided or dollars earned, and for a confidence interval showing the plausible range of the true effect.

Relative and absolute changes

Relative changes can sound dramatic while absolute changes are small. A defect rate falling from 2% to 1% is a 50% relative reduction but a one percentage point absolute reduction: out of 1,000 parts, ten fewer defects. Both are correct; the absolute figure usually matters more for decisions.

Common biases and traps

Survivorship bias: studying only successes, such as businesses that survived, and drawing lessons that ignore the failures who did the same things.
Selection bias: comparing groups that differed from the start.
Publication bias: positive results are more likely to be published or promoted than null results, so the published record overstates effects.
Cherry-picking: reporting the favourable outcome measures, time periods or sites and omitting the rest.
Moving goalposts: changing the success measure after seeing the data.
Overgeneralisation: applying results from one context to very different ones.
Authority bias: trusting a claim because of who made it rather than the evidence behind it.

Strength of evidence

Some kinds of evidence are generally stronger than others for questions of cause and effect. Well-designed systematic reviews and meta-analyses, which combine many studies using transparent methods, are usually stronger than single studies. Randomised controlled trials are usually stronger than observational comparisons, which are stronger than case studies and expert opinion. This ordering is a guide, not a rule: a large, careful observational study can be more useful than a small, poorly run trial, and case studies are valuable for understanding how and why things work.

Applying results to your business

Even strong evidence must be applied with judgement. Ask:

Were the conditions of the study similar to yours: industry, size, workforce, customers, regulations and technology?
What made the intervention work there, and do those conditions exist in your business?
What would it cost to try, and could a small trial test it locally?

Where evidence is promising but uncertain, a local pilot with clear measures is often the most practical way to find out.

Surveys reported in the media

Surveys are among the most quoted forms of research and among the easiest to misread. When a headline says that most businesses or customers believe something, check who was surveyed, how they were recruited, how many responded and the exact wording of the question. Online opt-in polls, surveys of a company’s own mailing list and surveys commissioned to support a product launch often produce striking results that would not hold in a representative sample. Questions that suggest an answer, or that offer only favourable options, shape the result. A survey that does not publish its question wording and sample details should be treated as an opinion piece rather than evidence.

Reading charts critically

Charts can mislead as easily as words. A vertical axis that does not start at zero exaggerates small differences. Different scales on two charts placed side by side make unlike changes look alike. Selected time periods can show a rise or a fall from the same data. Cumulative charts always rise, hiding slowdowns. Three-dimensional effects distort proportions. Before drawing conclusions from a chart, read the axis labels and scales, check the time period and ask what the chart would look like with a different starting point or the full data set.

Expert opinion and consensus

Expert judgement is valuable, particularly where research is limited, but individual experts can be wrong, overconfident or influenced by their own interests. The views of a broad group of independent experts, especially when expressed through professional bodies, standards or formal reviews, are more reliable than any single voice. When experts disagree, understanding why, whether because of different evidence, different assumptions or different values, is more useful than counting heads.

When good evidence is missing

For many business questions, no strong study exists. Decisions must then rest on weaker evidence, reasoning from principles, the experience of comparable businesses and small local tests. The honest approach is to say so: state that evidence is limited, make the decision reversible where possible and plan to learn from what happens.

Reviewing a body of research

Sometimes the task is to understand what is known about a topic, not to judge a single study. A practical literature review for a business decision:

defines the question and the sources to search;
records each relevant study with its key features and findings, often in an evidence table;
groups findings by theme or question rather than listing studies one by one;
notes where studies agree, disagree or are silent, and why; and
reaches a judgement about the overall strength of evidence.

When summarising sources, use your own words and cite the source, rather than copying passages. Accurate paraphrasing shows understanding and avoids misrepresenting or plagiarising the original.

Summarising for decision-makers

Decision-makers need the bottom line first: what the evidence suggests, how confident that conclusion is and what it means for the decision. A useful summary states the question, the main finding with its size and uncertainty, the quality of the evidence, important caveats and a recommendation. Detailed tables and references belong in an appendix. Overstating certainty to make a summary sound decisive does more harm than admitting what is unknown.

A worked example

This is an illustrative example. The vendor, study and figures are hypothetical.

A manufacturer is offered a condition-monitoring system. The vendor’s brochure states that “customers reduced unplanned downtime by 40%”. The operations manager asks for the underlying study.

The study covers three customer sites chosen by the vendor. Downtime was compared for the twelve months before and after installation, with no comparison sites. At two of the sites, the installation coincided with a broader maintenance overhaul and new staff. The average reduction was 40%, but it ranged from 10% to 65%, and one site’s “before” year included an unusually long breakdown. There is no independent replication.

The manager concludes that the evidence suggests the system may help but does not establish how much. The manager proposes a six-month trial on two of the plant’s most critical machines, comparing their downtime with similar machines without the system, and agrees success measures with the vendor before starting. The trial shows a reduction in unplanned stoppages on the monitored machines, smaller than claimed but enough to justify a staged rollout.

How this applies to a small Australian business

Read methods and results before conclusions.
Ask who funded the research and whether data and methods are available.
Look for a credible comparison before accepting claims of improvement.
Focus on absolute effects and their uncertainty.
Judge whether conditions match your business.
Use small local trials to test promising but uncertain claims.
Check survey wording and samples before quoting survey results.
Read chart axes and time periods before drawing conclusions.
Make decisions reversible where good evidence is missing.
Summarise evidence honestly, with confidence levels stated.

Common mistakes

Accepting percentages without baselines.
Treating association as causation.
Relying on a single, unreplicated study.
Ignoring survivorship and selection bias.
Confusing statistical with practical significance.
Copying text from sources instead of summarising in your own words.
Quoting opt-in polls as if they represented a whole market.
Being misled by truncated chart axes or selected time periods.
Overstating certainty in summaries for decision-makers.

Frequently asked questions

Does peer review guarantee a study is correct? No. It reduces some errors, but reviewed studies can still be flawed, and replication by independent researchers is stronger evidence.

What is a confidence interval? A range of values that is likely to contain the true effect, given the data. A wide interval means the estimate is uncertain.

Are case studies useless as evidence? No. They are valuable for understanding how and why something works in context, but they cannot on their own show how often it works or how large its effect is.

How can a vendor’s claims be tested cheaply? Ask for the data behind the claim, speak to customers not chosen by the vendor and, where possible, run a small trial with agreed measures and a comparison.

What is regression to the mean? When something is measured after an unusually extreme result, the next measurement tends to be closer to average, even without any intervention. Improvements measured after a bad period can therefore look larger than they are unless a comparison group is used.

Questions to ask

What exactly was studied, and how were the participants or sites chosen?
What was the comparison, and what else changed at the same time?
How large is the effect in absolute terms, and how uncertain?
Who funded the work, and are the data and methods available?
How strong is the evidence overall, and how reversible is our decision if it proves wrong?
Would the conditions of the study apply in our business?

Bringing it together

Reading research critically means asking consistent questions about the question, method, sample, comparison, effect size, uncertainty, bias and relevance. Statistical significance is not practical significance, relative changes can mislead, correlation is not causation and single unreplicated studies deserve caution. Reviewing a body of research by theme, summarising in your own words and presenting the bottom line with honest confidence levels turn evidence into decisions. The thinking in probabilities article shows how to weigh evidence as degrees of belief.

Source: KEVOS editorial notes. Examples and figures in this article are illustrations.