Evaluation and action research inside organisations: judging what worked and improving as you go (Research and Evidence for Better Decisions) | READOS

Organisations launch initiatives all the time: a new onboarding program, a safety campaign, a revised quoting process, a customer service standard, a lean improvement program, a new software system. Most are judged informally. If people seem happy and nothing obviously goes wrong, the initiative is declared a success; if a vocal critic complains, it may be quietly dropped. Rarely does anyone ask, in a structured way, whether the initiative achieved what it was meant to, at what cost, for whom, with what side effects, and what should change.

Evaluation answers those questions. It applies research methods to judge the merit and value of programs, projects and changes, using clear questions, defined measures, baselines and comparisons. Action research is a related approach in which the people doing the work study and improve their own practice through repeated cycles of planning, acting, observing and reflecting, combining improvement with learning. Both turn activity into evidence and help organisations invest in what works.

This article explains the purposes and types of evaluation, how to build a logic model, choose evaluation questions and indicators, establish baselines and comparisons, use mixed methods, look for unintended effects, consider costs and decide who evaluates. It then explains action research, how its cycles work, how to keep it rigorous and how evaluation and action research fit with everyday improvement. It is part of a series on research and evidence for business decisions.

Why evaluate

Evaluation serves several purposes:

Accountability: showing whether resources achieved what was promised.
Improvement: finding what works, what does not and how to do better.
Decision-making: deciding whether to continue, expand, change or stop.
Learning: building knowledge that informs future initiatives.

Being clear about the main purpose shapes the design. An evaluation for improvement emphasises timely, practical feedback; one for a continue-or-stop decision emphasises robust evidence of outcomes and costs.

Types of evaluation

Formative evaluation happens during implementation, providing feedback to improve the initiative while it is running.
Summative evaluation happens at or after the end, judging overall results.
Process evaluation examines how the initiative was delivered: whether it reached the intended people, was implemented as planned and what helped or hindered it.
Outcome evaluation examines whether intended changes occurred, such as fewer errors, faster onboarding or higher customer satisfaction.
Impact evaluation examines whether the initiative caused those changes, compared with what would have happened without it.

Process evaluation is often neglected, yet it explains outcomes. An initiative that “failed” may never have been delivered as designed.

Logic models

A logic model, sometimes called a program logic, sets out how an initiative is expected to work:

Inputs: resources such as people, time, money and equipment.
Activities: what is done, such as training sessions, new procedures or system changes.
Outputs: direct products of activities, such as people trained or procedures issued.
Outcomes: changes that result, in the short and medium term, such as improved skills, changed behaviour or fewer defects.
Impact: longer-term results, such as higher profit, better safety records or customer retention.

The logic model makes assumptions visible: why should training lead to changed behaviour, and why should that reduce defects? Each link is a hypothesis the evaluation can test. It also distinguishes outputs from outcomes, a common confusion: training 50 people is an output; whether they work differently is an outcome.

Evaluation questions

Good evaluations focus on a few clear questions, often grouped as:

Effectiveness: did the initiative achieve its intended outcomes?
Efficiency: were the outcomes achieved at reasonable cost and effort?
Appropriateness: did it address the right problem in a suitable way?
Equity: did it benefit the intended groups, and were some left out?
Sustainability: will the benefits last after the initial effort ends?

Questions should be agreed with the people who will use the evaluation, before data collection begins.

Indicators and baselines

Indicators are measures that show whether outcomes are being achieved: time for new staff to work independently, error rates, retention after six months, complaint numbers, cycle times. Good indicators are relevant, measurable with available or collectable data, sensitive to change and hard to game.

A baseline measures indicators before the initiative starts, so change can be assessed. Without a baseline, claims of improvement rest on memory and impression. Where baselines were not collected, existing records may allow them to be reconstructed.

Comparison and attribution

Changes in indicators may result from the initiative, from other changes happening at the same time or from natural fluctuations. Stronger evaluations compare the group that received the initiative with a similar group that did not, use staged rollouts that create natural comparisons or compare trends before and after against trends elsewhere. Where no comparison is possible, evaluations should acknowledge that other factors may explain results. The benefits move after approval article discusses how expected benefits change during and after delivery.

Mixed methods

Most evaluations combine numbers and narrative. Quantitative indicators show what changed and by how much; interviews, observation and open-ended feedback explain why and how, reveal implementation problems and capture effects not anticipated in the indicators. Combining them gives both credibility and understanding.

Unintended effects

Initiatives often have effects nobody planned: a target that improves one measure at the expense of another, a procedure that adds workload elsewhere, a system that frustrates customers in ways not measured. Evaluations should actively look for unintended consequences, positive and negative, by asking open questions and checking related measures.

Costs and value

An outcome is worth more or less depending on what it cost. Recording the full costs of an initiative, including staff time, and comparing them with the value of outcomes, or with the cost of alternative ways to achieve them, supports decisions about whether to continue or expand.

Evaluating training

Training is one of the most commonly evaluated initiatives, and one of the most often evaluated poorly. A widely used framework, associated with Donald Kirkpatrick, distinguishes four levels: reaction, whether participants found the training useful; learning, whether they gained knowledge or skill; behaviour, whether they apply it at work; and results, whether the organisation’s outcomes improved. Many evaluations stop at reaction forms completed at the end of a session, which say little about whether anything changed. Checking skills with practical assessments, observing work weeks later and tracking related performance measures give a far better picture. Behaviour often fails to change because of conditions back at work, such as lack of time, tools or support, which process evaluation can reveal.

Evaluating new systems and processes

New software, equipment and procedures are usually justified by expected benefits, such as time saved, errors avoided or faster response. Evaluating them means measuring those benefits after implementation against the baseline, checking whether the system is actually used as intended and looking for new problems it has introduced, such as workarounds or extra data entry. Evaluations are most useful when planned before go-live, so baselines exist, and repeated after the system has settled, because early results often reflect learning curves rather than the system’s long-term value.

Ethics in internal evaluation

Evaluations inside organisations involve colleagues, and their findings can affect people’s roles and reputations. Participants should know the purpose of the evaluation, how information will be used and whether comments will be attributed. Findings should focus on systems, designs and conditions rather than individual blame, unless performance issues are being handled through proper processes. Reporting negative results honestly, while treating people fairly, builds the trust that future evaluations depend on.

Who evaluates

Internal evaluators understand the organisation and can provide timely feedback, but they may be seen as biased or may find it hard to criticise colleagues. External evaluators bring independence and specialist skills but cost more and need time to understand the context. Many organisations combine the two: internal staff collect data and an external reviewer checks the design and conclusions.

Action research

Action research is an approach in which practitioners study and improve their own practice. Instead of an outside researcher studying a team, the team investigates a problem in its own work, tries changes, gathers evidence about their effects, reflects and adjusts. It typically proceeds in cycles:

Plan: understand the problem and decide on a change to try.
Act: implement the change.
Observe: collect evidence about what happens.
Reflect: interpret the evidence and decide what to do next.

The cycle will look familiar to anyone who knows the plan-do-check-act cycle used in quality improvement, described in the A3 problem solving and PDCA article. Action research emphasises participation, learning and the development of practical knowledge, and it suits problems where the people involved hold much of the knowledge and must own the solution.

Keeping action research rigorous

Because action researchers are evaluating their own work, rigour requires deliberate effort:

Document each cycle: what was planned, what was done, what was observed and what was concluded.
Use more than one source of evidence, such as data, observation and feedback from others.
Invite challenge from colleagues or an external critical friend.
Reflect on bias: practitioners’ hopes for a change can colour how they see evidence.
Share findings openly, including what did not work.

From evaluation to use

Evaluations are only valuable if they are used. Reports should answer the agreed questions clearly, with recommendations linked to evidence, and be timed to inform decisions. Discussing findings with the people responsible for the initiative, and agreeing actions, makes use more likely. The delivered is not adopted article explains why changes often stall after go-live and how evaluation evidence helps them stick.

A worked example

This is an illustrative example. The business and figures are hypothetical.

A manufacturer introduces a structured onboarding program for production staff: a buddy system, a skills checklist and weekly check-ins during the first two months. Before launch, the operations manager builds a logic model linking the activities to faster competence, fewer early errors and better retention, and records baselines from the previous year: new starters took an average of about nine weeks to work unsupervised, and about one in four left within six months.

The evaluation combines records and interviews. After a year, new starters reach unsupervised work in about six weeks on average, and about one in seven leave within six months. A comparison with a sister site that did not adopt the program shows little change over the same period, strengthening the case that the program caused the improvement. Process evaluation finds that the weekly check-ins were often skipped during busy periods, and interviews reveal that buddies felt the role added to their workload without recognition.

The site then runs two action research cycles with team leaders and buddies. They trial protected time for check-ins and a small allowance for buddies, observe completion rates and feedback, and refine the checklist. Check-in completion rises, and the program is extended to the warehouse.

How this applies to a small Australian business

Build a simple logic model before launching an initiative.
Agree evaluation questions with the people who will use the answers.
Record baselines before changes start.
Look for comparison groups or staged rollouts.
Evaluate delivery as well as outcomes.
Ask about unintended effects.
Use action research cycles for problems that teams must solve themselves.
Evaluate training beyond reaction forms, checking skills, behaviour and results.
Plan system evaluations before go-live so baselines exist.
Act on findings and record what changed.

Common mistakes

Counting outputs as outcomes.
Starting without a baseline.
Attributing all change to the initiative without a comparison.
Ignoring how the initiative was actually delivered.
Evaluating too late to inform decisions.
Judging training by end-of-session feedback alone.
Using evaluations to blame individuals for system problems.
Action research without documentation or challenge.

Frequently asked questions

How much should evaluation cost? In proportion to the initiative and the decisions at stake. Many small evaluations need little more than baselines, existing records and a few interviews.

Is action research the same as continuous improvement? They are closely related. Action research places more emphasis on documenting learning and on the participants as researchers, but both use iterative cycles of change and evidence.

What if an evaluation shows an initiative did not work? That is valuable evidence. Understanding why, through process evaluation, often shows whether to fix, redesign or stop it.

Who should see evaluation results? The people who commissioned and delivered the initiative, those affected by it and decision-makers, presented in a way that supports learning rather than blame.

When should an evaluation be planned? At the same time as the initiative itself, so that baselines are recorded, indicators are agreed and data collection is built into delivery rather than added afterwards.

Questions to ask

What outcomes is this initiative meant to achieve, and how will we know?
Do we have baselines and a comparison?
Was the initiative delivered as planned?
What unintended effects have occurred?
Are we judging new systems by how well they are used, not just by whether they went live?
How will we know whether training changes what people do at work?
How will findings feed into decisions and improvements?

Bringing it together

Evaluation applies research methods to judge whether initiatives achieve their purposes, at what cost and with what side effects. Logic models make assumptions explicit, clear questions and indicators focus effort, baselines and comparisons support attribution, and mixed methods explain results. Action research lets teams study and improve their own work through documented cycles of planning, acting, observing and reflecting. Used together and acted on, they help organisations invest in what works and learn from what does not.

Source: KEVOS editorial notes. Examples and figures in this article are illustrations.