Validity and Reliability
One slide, two terms, one definition. This page sets out exactly what the material says about each, the dependency it creates between them, and the four related quality terms it names and never explains.
What the slide actually says
The supplied teaching source addresses both terms on a single slide, titled "Validity and reliability". It is quoted here in full, because the difference in how the two terms are handled is the substance of the page.
TWO TERMS, TWO STANDARDS OF TREATMENT
| Validity | Reliability | |
|---|---|---|
| General definition | None given | "the extent to which a research instrument… will produce the same results if the research was repeated" |
| What is supplied instead | A single worked example about measuring student turnover | A definition, plus an example of an instrument (a survey) |
| Scope statement | None — the slide never says which kinds of research it applies to | "more application to quantitative than qualitative research" |
| Procedure for achieving it | None given | None given |
| Test for whether you have it | None given | None given |
Both terms are treated on the same slide. The asymmetry is a property of the supplied material.
Validity is given an example in place of a definition
The sentence opens "Validity means, for example…" and never returns to complete the definition it started. What follows is one instance: a project designed to measure levels of student turnover in an institution must actually do that.
You can generalise from the instance — a project is valid when it does what it was designed to do — but be clear that the generalisation is yours, not the source's. If you are writing a methodology chapter and need a definition of validity you can cite, this material does not contain one.
The example is still worth something. It states the test in the right direction: you start from what the project was designed to measure and ask whether it did that, rather than starting from the data and asking whether it looks reasonable. Most validity failures in project research are of that shape — an instrument that measures satisfaction with a process and a claim about the effectiveness of the process.
Reliability, and the scope statement attached to it
Reliability is the one term in this area the material defines cleanly. Note what the definition contains: an instrument, a repetition, and the sameness of results. Note also what it does not contain — any statement about whether those results are correct.
What the definition covers
- The unit is the instrument, not the study — "a research instrument, such as a survey"
- The test is repetition — "if the research was repeated"
- The criterion is sameness — "produce the same results"
- A survey is offered as the example of an instrument, with no length, response rate or sample size attached
What the definition does not cover
- Whether the same results are the right results
- How many repetitions, or over what interval
- How much difference between repetitions is tolerable
- Any procedure or statistic for establishing reliability
The scope statement — that reliability "has more application to quantitative than qualitative research" — is comparative and hedged. It does not say reliability is inapplicable to qualitative work, only that it applies less. The material's fuller treatment of what replaces it in qualitative work appears in a later week and is set out on Validity Criteria for Qualitative Research.
The dependency the slide creates, and where it breaks
The validity example does something consequential: it defines validity partly in terms of reliability. To be valid, the project "must find a reliable way of measuring" the thing. Follow that through against the slide's own scope statement and against the rest of the week, and the chain does not hold.
- Validity requires a reliable measure
- Reliability applies more to quantitative work
- So validity applies less to qualitative work
- But every project is required to be valid, reliable or trustworthy
THREE DIFFERENT STATUSES GIVEN TO THE SAME TWO WORDS
| Where it appears | Status given to validity and reliability | What that implies |
|---|---|---|
| The validity and reliability slide | Properties a project must achieve — it "must find a reliable way of measuring" | They are outcomes of design work and can fail |
| The method-selection slide | Open questions to ask of any project before choosing a method | They are selection criteria applying to all approaches equally |
| The quantitative and qualitative comparison table | Listed as intrinsic features of quantitative methodology, alongside "reproducible", "objective" and "generalizable" — the source's own spelling | They are attributes a paradigm confers rather than results a project earns |
All three appear in the same week. The third is quoted from a published guide, which may explain the difference in framing but does not remove it.
Four related terms that are named and never explained
The quality vocabulary in this material is wider than the two terms on the slide, and the rest of it is unsupported. The count below is the count of the list as printed here, drawn from terms used in the material without definition. Rigour, the sixth term in this family, is treated on The Context of Data Gathering, where it is defined by way of two further terms.
Named in the material, defined nowhere in it
- Trustworthiness
- Appears once, in the method-selection question "will the research be considered valid, reliable or trustworthy when reviewed?". No definition, no criteria, no cross-reference. The later qualitative week supplies four alternative criteria but never uses the word trustworthiness to unify them.
- Generalisability
- Appears once, in the method-selection question about "the ways in which you hope to generalize any specific results from one setting to a larger number of settings". The question assumes the reader already knows what generalisability is; the material never says.
- Internal validity
- Named in the later qualitative week as one of the traditional criteria for judging quantitative research, paired against credibility. Not defined anywhere in the supplied material, and the general treatment of validity never distinguishes internal from external.
- External validity
- Named in the same table, paired against transferability. Not defined anywhere in the supplied material, and the same week separately states that there is no point in trying to establish validity in any external sense.
- Soundness
- Used in the material's definition of rigour — "establishing the soundness and the dependability of their research" — and never defined. Rigour is therefore defined by way of two further terms, one of which is undefined and one of which is defined differently in a later week.
Using the two criteria on a real project brief
The material states the criteria and supplies no procedure for either. The sequence below applies only what the source says — a project must do what it was designed to do, and an instrument should produce the same results on repetition — and is offered as synthesis rather than as the source's method. For how published project management research actually words these claims, see Rigour, Triangulation and Validity in Practice.
Four checks you can run before collection starts
Write the claim you intend to make
One sentence, in the form your report will use it. "Delivery capability improved after the change" or "schedule slippage is driven by late scope". This is the thing validity is assessed against.
Write what your instrument actually captures
Also one sentence, and deliberately literal. "Respondents' agreement with statements about how the change felt." Put the two sentences side by side. The gap between them is your validity risk, stated in your own words.
Ask what would change the answer on a second run
Different respondents, a different week, a different interviewer, a different set of documents. The source's test is sameness on repetition, so name the sources of variation that repetition would expose.
Decide which criterion your approach can carry
If the work is qualitative, the material itself says reliability applies less, and points to a different criteria set. Say which set you are using in the methodology chapter and cite where the definitions came from, because this material does not supply them.
What to carry forward
- Reliability, as this material defines it, is sameness of results on repetition of an instrument. Nothing in the definition speaks to correctness.
- Validity is illustrated and never defined here. Use the example's direction of travel — did the project do what it was designed to do — and cite someone else for the definition.
- The slide makes validity depend on reliability and then limits reliability to quantitative work. The material does not resolve what that means for qualitative projects.
- Trustworthiness, generalisability, internal and external validity and soundness are all named in the material and none is defined.
- The whole treatment is a single slide. Budget for a second source at proposal stage, not at examination.
Frequently asked questions
How does this material define validity?
It does not. The slide begins "Validity means, for example…" and supplies a single instance — a project designed to measure student turnover must actually do that. The general form is left for the reader to infer, so if you need a definition to cite, take it from a methods text rather than from this material.
How does it define reliability?
As the extent to which a research instrument, such as a survey, will produce the same results if the research were repeated. That is a clean general definition, and it is silent on whether the repeated results are correct — an instrument can be consistently wrong and still satisfy it.
Does reliability apply to qualitative research?
The material says only that the issue has more application to quantitative than qualitative research, which is comparative rather than exclusionary. Elsewhere it goes further, saying it would not be meaningful to test whether qualitative collection methods yield consistent results across contexts, and a later week substitutes a different set of criteria.
What is trustworthiness in this material?
An undefined term. It appears once, in a method-selection question asking whether the research will be considered valid, reliable or trustworthy when reviewed, and receives no definition, criteria or cross-reference anywhere. The later qualitative week's four criteria are never gathered under that heading.
Can a project be reliable but not valid?
On the definitions this material supplies, yes — reliability is sameness on repetition and says nothing about whether the right thing is being measured. The material does not state this consequence, but it follows directly from the wording of the reliability definition.
What should I do about the gap in coverage?
Name it and fill it deliberately. Decide which quality criteria your approach uses, take citable definitions from a methods text, and state in your methodology chapter which definitions you adopted. The one place this material is complete is the qualitative alternative set, where all four criteria are named and defined.
References and source attribution
- O'Leary, Z. 2010, The essential guide to doing your research project, Sage Publications. (The source of the comparison table that lists reliability and validity as features of quantitative methodology, p. 105.)
- Bryman, A. 2016, Social Research Methods, 5th ed., Oxford University Press, Oxford.
- Trochim, W. M. K. 2006, Research Methods Knowledge Base (http://www.socialresearchmethods.net/kb/probform.php). (The knowledge base the material points to for its qualitative validity criteria.)
- Veal, A. J. 2005, Business Research Methods: A Managerial Approach, Longman.
- The supplied teaching source: consolidated weekly slide material and study notes on research design and on qualitative research, used here as the basis for every statement attributed to "the source", "the week" or "the material".
Suggested questions for Ask KEVOS
- Compare the claim I want to make with what my instrument actually measures, and tell me where the gap is.
- Which quality criteria should my project be judged against, given that it is qualitative?
- Draft the paragraph of my methodology chapter that addresses validity, and mark where I need an external citation.
- What would change my results on a second run, and which of those can I control?
- Explain the difference between the quantitative criteria and the qualitative alternatives as this material presents them.