Evaluating an Article Before You Cite It
Ten questions turn a stack of downloaded PDFs into a set of usable positions. Answer them as you read, or you will be re-reading everything in three months.
Two passes, not one
The supplied source separates reading into two distinct movements, and the separation is the point. "Read the literature quickly (skim read) and make some short notes as to its relevance and quality." Then: "Take notes as you read the literature."
The first pass answers one question — does this belong in my review at all? The second pass answers the ten questions below. Merging them means reading everything at full depth, including the third of your pile that will turn out to be irrelevant.
The ten questions
THE TEN READING QUESTIONS, AND WHAT EACH ONE IS FOR
| The source's question | What the answer lets you do |
|---|---|
| Is it a general textbook or article, or does it analyse a specific issue? | Decides whether the work is background or evidence. A textbook establishes accepted position; a focused study can be argued with. |
| Is the article relevant to your industry? | Governs how far you can transfer its conclusions without qualification. Relevance is not a yes/no — record the distance. |
| Is it an empirical report, a dissertation, a theoretical study, a sociological or political account, a historical overview — all or some of these? | Tells you what kind of claim the work is entitled to make. A theoretical study cannot evidence prevalence; an empirical report cannot settle a definition. |
| Who is the author, and do they have an academic or industry background implying knowledge and subject integrity? | One of the eight credibility questions, reused here at reading depth. Affiliation is weak evidence on its own but useful in combination. |
| Does it follow a particular school of thought? | Lets you group works into camps. A review organised by camp is far stronger than one organised by publication date. |
| What is its theoretical basis? Is it well researched? | Surfaces the assumptions the work inherits. Two studies with opposite findings often differ in their theory, not their data. |
| What definitions does it use? | The most under-used of the ten. Studies that appear to disagree frequently define the central construct differently, and that is a finding in itself. |
| What is its general methodological approach? What methods are used? | Determines whether the work is comparable with others in your review, and whether its design could be reused for yours. |
| What kinds of data does it use to back up its argument? | Separates assertion from evidence. Some published claims rest on nothing you can inspect. |
| What conclusions does it come to? | Last, deliberately. A conclusion means little until you know the approach, the definitions and the data that produced it. |
The ten questions are the source's; the second column is a synthesis for this page explaining what each answer buys you.
The four questions that do most of the work
Questions five to eight — school of thought, theoretical basis, definitions and methodological approach — are the ones that let you compare studies rather than merely list them. Everything the source says about writing a review depends on being able to make those comparisons.
Why the middle four matter most
Turns a list into camps
Once you can name the position a work argues from, you can group works by position. That grouping is the skeleton of a thematic review and the thing that separates a review from an annotated bibliography.
Explains disagreement
When two credible studies reach opposite conclusions, the explanation is more often a difference in underlying theory than an error in either. Recording the theory means you can say why they differ.
Finds the invisible conflict
Two studies of 'project success' may be measuring different things entirely. A review that notices this has found a real gap; one that does not will report a false consensus or a false contradiction.
Makes works commensurable
You cannot weigh a survey of seventy-one respondents against a two-case study without knowing that is what you are doing. Method is also where you find designs worth borrowing.
Where this sits against credibility screening
The supplied source contains two separate question sets that are easy to confuse. The eight credibility questions decide whether a source is admissible at all. The ten reading questions characterise a source you have already admitted.
TWO QUESTION SETS, TWO DIFFERENT JOBS
| Credibility screening (eight questions) | Article evaluation (ten questions) | |
|---|---|---|
| When | Before you invest reading time | During the second, careful read |
| Decides | Whether the source is admissible | How the source can be used and grouped |
| Typical output | In or out | A characterisation you can compare with others |
| Covered on | Assessing the academic value of sources | This page |
Both sets are in the supplied source. Running the ten before the eight means reading unusable material carefully.
A working note template
The source asks you to take notes but specifies no format. The template below is a synthesis for this page — it exists so that every work in your pile is characterised on the same axes, which is what makes them comparable later.
One record per work
- Full bibliographic details, captured the moment you find the work, not later
- Type — empirical, theoretical, review, dissertation, historical, or a combination
- Field and industry, plus how far that sits from yours
- School of thought or position argued from
- The central construct's definition, quoted verbatim with a page number
- Methodology and methods, including sample and how it was obtained
- What kind of data backs the argument, and whether you can inspect it
- Conclusions, and any limitation the authors state themselves
- Your own one-line judgement of relevance and quality from the skim pass
From notes to review
Answering the ten questions consistently produces something that can be sorted. The sorting is the beginning of the review: works that share a school of thought group together, works that define the construct differently form a contrast, and works that used a method you could adopt go into your methodology chapter rather than your review.
What to do with the grouped material next is set out in Critical summary and paraphrasing and Writing the literature review.
What to carry forward
- Two passes: skim for relevance and quality, then read for the ten questions.
- Approach and definitions come before findings. A conclusion is uninterpretable without them.
- School of thought, theory, definitions and method are the four answers that let you compare works rather than list them.
- Copy central definitions verbatim with a page number. Paraphrased definitions drift and hide real disagreements.
- The ten questions omit limitations. Record them anyway — the source's own annotation anatomy requires them.
Frequently asked questions
Do I have to answer all ten questions for every source?
For anything you intend to cite, yes — that is what makes your sources comparable. For material you are still triaging, the skim pass answering relevance and quality is enough, and the source explicitly separates the two passes.
What is the difference between the ten reading questions and the eight credibility questions?
The eight credibility questions decide whether a source is admissible before you invest reading time. The ten reading questions characterise a source you have already admitted, so you can group and compare it. Both sets are in the supplied source and they run at different moments.
Why does the source put conclusions last?
Because a conclusion is only interpretable once you know the approach, the theoretical basis, the definitions and the data that produced it. Reading for conclusions first is how reviews end up reporting contradictions that are actually definitional differences.
How do I judge whether a work is 'well researched'?
The source asks the question but sets no criteria. In practice the answerable proxies are whether the data is described well enough to inspect, whether the sample and how it was obtained are stated, and whether the authors name their own limitations.
Should I note limitations even though the ten questions do not ask for them?
Yes. The omission is an inconsistency in the supplied material — the same source's annotation anatomy makes limitations a required element, and every examined work surveyed for this library states its own.
References and source attribution
- Supplied teaching source, Week 4: Reviewing the Literature — study notes and slide deck.
- Bryman, A. 2016, Social Research Methods, 5th ed., Oxford University Press, Oxford.
- O'Leary, Z. 2017, The Essential Guide to Doing Your Research Project, 3rd ed., Sage Publications, London.
- Naoum, S. G. 2013, Dissertation Research & Writing for Construction Students, 3rd ed., Routledge.
Suggested questions for Ask KEVOS
- Turn the ten reading questions into a note template I can fill in for each paper.
- How do I tell whether two studies disagree or just define their terms differently?
- What counts as evidence I can inspect in a research paper?
- How should I group papers once I have characterised them?
- What should I do with a paper that is credible but from a different industry?