A Worked Systematic Literature Review
The apparatus is all here: a twenty-stage process diagram, twelve printed search strings and counts at every stage that produced records. What is missing is the search date, the list of retained papers, and one sentence about what a review's criteria actually govern.
What a worked review looks like when someone prints the apparatus
This library records inclusion and exclusion criteria being named in a proposal component list and defined nowhere, and it records search strategy taught as nine database steps with no worked instance attached. One examined master's work — called Work K here — supplies the instance. It ran a systematic quantitative literature review, printed its search table and its criteria, justified each exclusion, and reported counts at every stage that produced records. Set it beside literature search strategies and the five-step development process, neither of which has a worked flow attached.
Everything below is one work's practice. One project is not a population: none of these counts is a benchmark, and the defects are defects in a document, not in a researcher. The thesis was examined and passed.
The search: four databases, one engine, twelve strings
Four scholarly databases are named and one — a general academic search engine — is designated the prime engine on the stated ground that it aggregates the others. Every result count in the thesis is attributed to that one engine. The other three appear once each, in the sentence that names them: no count, no date and no search string is reported for any of them.
The first criterion is a terminology search rather than a screening stage. Twelve search strings are printed with their result counts, and the counts are used to make a vocabulary decision: one candidate term returned 15,300 results against 9,170 for its near-synonym, so the more common term was carried into every later combination. That is a decision about which word the field uses, not a finding about the field, and a page carrying it should say which of the two it is.
THE THREE CRITERIA, AND WHAT EACH ONE ACTUALLY DID
| Criterion | What it was | Records returned | Excluded | Retained |
|---|---|---|---|---|
| 1 | Terminology test — twelve strings, twelve counts, printed as a table | 12 counts, 5 to 15,300 | No screening stage | None enter the review |
| 2 | Five-keyword combination on the topic | 27 | 27 | 0 |
| 3 | Three-keyword combination, constraints reduced after criterion 2 failed | 321 | 288 | 33 |
| Citation chasing | Backward chasing from relevant papers — performed, named, and declared not to create a new criterion | Not counted | Not counted | 5 named publications |
Counts are this study's own, from one search engine, on an unstated date. They are not norms and cannot be reproduced.
One thing in the search reporting is worth copying outright: the thesis names five excluded papers individually, by author and year, with the reason each was rejected. That costs about seventy words, and it is the only place in this library where a reviewer shows exclusion reasoning on individual records.
Two named classes of false positive — and this is the most transferable page in the batch
The keyword is only in the reference list
A retrieved paper matches because a work it cites carries the term in its title. The thesis gives a worked example: a 2021 paper whose only connection to the topic is one citation.
The keyword is an ordinary English word
The thesis quotes two hits verbatim, where the topic term appears in phrases about being "the first line of defence when building a solution" and about "the defence of area".
Screen in two passes, and say so
The one screening procedure the thesis describes is correct: title and abstract sufficed to exclude some records, others had to be read whole. Two-stage screening, attributed, in a sentence.
The criteria, and the justification section that goes with them
The criteria table carries five inclusion rules against six exclusion rules. The first four pairs are exact logical mirrors: dates, language, full-text availability, and topic focus. Exclusion criteria 5 and 6 have no inclusion counterpart, and inclusion criterion 5 — the list of five admissible publication types — has no exclusion counterpart.
THE SIX EXCLUSION CRITERIA AND THEIR WRITTEN JUSTIFICATIONS
| # | The rule | The stated reason |
|---|---|---|
| 1 | Studies published before 2012 | Technology and its delivery methods evolve; work older than the current decade is treated as irrelevant to the setting studied |
| 2 | Studies in languages other than English | Translation is "not impossible", but the trade-off between time and effort versus yield justifies the restriction |
| 3 | Not available in full text or not retrievable online | Others could not reproduce or validate findings without access to the source material |
| 4 | Subject does not focus on the topic | Potential for misdirection |
| 5 | Affiliated to a public organisation or industry partner that could introduce bias | Sponsored publications are susceptible to the funding effect. The only justification with a citation attached |
| 6 | Literature reviews | The thesis "aims to gather data from source materials only in an effort to reduce any risk associated with improper analysis" |
Six criteria, six written justifications, one of them cited. Nothing in fourteen weeks of the supplied teaching material asks for one.
One further rule is applied and never declared. Criterion 3's 288 exclusions were made partly because papers "had insufficient referencing (>10 references)" and partly because they "lacked citing elsewhere". Neither rule appears in the criteria table. The reference threshold is the only numeric quality threshold anywhere in this library, it is unattributed and unjustified, and it may be used only as one researcher's private rule — never as a standard.
The flow, and it reconciles
- Criterion 2: 27 returned
- 27 excluded, 0 retained
- Criterion 3: 321 returned
- 288 excluded
- 33 retained
27 + 321 = 348 screened. 27 + 288 = 315 excluded. 348 − 315 = 33 retained. The thesis states none of those three batch totals, and every stage it does state is internally consistent: 288 excluded is stated in one section and 33 retained is stated thirty lines later in another, and neither is derived from the other. This is the only reconciling review flow in the library and an author may say so plainly.
Four criteria the review breaks in its own body
This is the sharpest finding in the work and every element of it is checkable against the printed reference list. It is also the cleanest teaching artefact this library holds on a distinction most reviews never make explicit: a criterion that bounds a review corpus is not a criterion that bounds a document.
THE FOUR BREAKS, AND WHAT EACH ONE COSTS
| Criterion | What the body does | Load carried by the excluded class |
|---|---|---|
| 1 — nothing before 2012 | Ten of seventy reference entries are pre-2012; two are cited in the body | The 2001 founding document of the whole method, cited five times — the thesis's most-cited source, and the origin of the values and principles its questionnaire is built on. Plus a 2004 introduction, cited three times, including the sentences that close the abstract and the Conclusions |
| 5 — no public-organisation or industry affiliation | Four such sources are cited ten times between them | A government strategic-update publication cited seven times, carrying the opening argument; a fact sheet that is the source of the gap argument; an organisational-structure page supplying the scope enumeration; and a guide that fixes the definition of the central term |
| 6 — no literature reviews | A 2012 decade-in-review is cited twice | Once to establish the field's history, once to support the claim that the topic remains relevant. The list also carries a systematic review and a paper on writing literature reviews, both uncited |
| 3 — full text, retrievable online | Five reference entries cannot be reached by any reader | Three give a local hard-drive file path; one web address points to an unrelated document by a different author from a different decade; one prints the bare word "Link" where the address belongs |
The criterion whose stated purpose is that other researchers can reach the sources is the criterion the reference list most visibly fails.
Where the method actually lives
The chapter headed "Research method" runs to 280 words — 2.6% of the thesis, the smallest method chapter in this library. What is in it is design rationale: why a systematic quantitative review, why a questionnaire, with two cited justifications. What is not in it is the method.
ONE METHOD, THREE CHAPTERS
| Component | Words | Which chapter it sits in |
|---|---|---|
| Design rationale | 280 | Research method |
| Search process, including the twelve-string table | 576 | Literature review |
| Review criteria table | 200 | Literature review |
| Exclusion criteria validation | 267 | Literature review — under the only unnumbered heading in the document |
| Participant eligibility, recruitment and sample description | 554 | A fourth chapter headed "Addressing the gap" |
| Total describing the method | 1,877 | The chapter named for it holds 14.9% of them |
Derived by counting the sections; the thesis states none of this.
The work also inverts the template it is written to, putting the method chapter before the literature review. That is not stylistic. Here the literature review chapter contains the search process, the criteria, the validation, the results summary and the review's findings — it is a results chapter, and a results chapter cannot precede the method that produced it. See eight examined theses compared.
Running one yourself
What this review did, and the four things it did not
- Print your search strings, your result counts and the date you ran them
- State whether your date bound was applied at the search or after it — the two produce different flows
- Name every database you searched, and report counts for each one you used
- Write a justification for each exclusion criterion, and cite the ones resting on a published claim
- Put every rule you actually applied in the criteria table, including any threshold you invented
- Report screened, excluded and retained at every stage, and make them add up
- State whether you removed duplicates, and how many (library addition)
- List the retained set so a reader can recover your output (library addition)
- Say who screened, and whether anyone checked (library addition)
- Say in one sentence that your criteria bound the review corpus, not the whole thesis (library addition)
- Count records found by citation chasing inside your flow, not beside it
What to carry forward
- Print the search strings, the counts and the date. Two of three are here; the missing one makes the other two unreproducible.
- A flow that reconciles is achievable and rare — this one does, at every stage that produced records.
- State that your criteria bound the review corpus and not the thesis, or your own citations will break them.
- The retained set is the artefact a review exists to produce. List it.
- Put every rule you applied in the criteria table, including the threshold you made up on the day.
- A chapter containing the search, the criteria and the findings is a results chapter, whatever the template calls it.
Frequently asked questions
What does it mean for a review's flow to reconcile?
It means the counts add up in both directions and every total appears in the document: records screened, records excluded and records retained. In this work, 27 plus 321 gives 348 screened, 27 plus 288 gives 315 excluded, and 348 minus 315 gives the 33 retained. The thesis prints the stage figures and never prints the batch totals, which is why they had to be derived.
Is it wrong to cite a source your own exclusion criteria would reject?
Not in itself. A review that could not cite the founding document of its own topic would be a worse document, and here that document is the most-cited source in the thesis. What is missing is one sentence saying that the criteria govern the papers admitted to the review, not every source cited elsewhere. Without it, four of six criteria appear broken in the body.
Why does the search date matter so much?
Because a general search engine's result counts change daily. This review prints twelve counts to four significant figures with no date attached, in a design chosen on the stated ground that following the same method should make results repeatable. Without the date, none of the twelve can be checked by anyone.
Do I need a second screener?
This work had none and does not say so, which is the point: single-handed screening is a limitation whether or not you can avoid it, and a reader can only judge it if you state it. If a second screener is not available, say the screening was single-handed and describe the two-stage procedure you used instead.
Should I count papers found by citation chasing?
Yes. This review performed backward citation chasing, named it, and explicitly declared that it did not create a new search criterion — the disclosure is exemplary and the decision is not. Five publications entered the analysis through it and appear nowhere in the counted flow, so the flow understates the corpus.
How long should the method chapter of a review be?
There is no standard here to quote. What this work shows is a consequence: its method chapter is 280 words, its actual method runs to about 1,877 words across three chapters, and a reader looking for the search strategy from the contents page will not find it. The length is less important than whether a reader can locate the method from the contents.
References and source attribution
- An examined master's work supplied as student work: a systematic quantitative literature review of one organisational setting, combined with a questionnaire, with its process diagram, search-string table, review criteria table and exclusion justifications printed. Researcher, supervisor, institution, jurisdiction and organisation scrubbed. Used as observed practice, not as a model answer.
- The published fifteen-stage method for systematic quantitative literature reviews named in the thesis's own process-diagram caption as the source it adapted. Cited here only as the comparator against which the twenty drawn stages were counted.
- The thesis's printed reference list of seventy entries, against which the four broken criteria, the thirty-two uncited entries and the five unreachable entries were verified individually.
- The supplied teaching source: weekly study notes, slide decks and assessment activities for a master's-level research methods subject in project management, in which inclusion and exclusion criteria are named in a proposal component list and defined nowhere. Author, institution and year not stated in the supplied files.
Suggested questions for Ask KEVOS
- How do I build a screening flow that reconciles from search to retained set?
- What should go in my inclusion and exclusion criteria table?
- How do I write a justification for each exclusion criterion?
- How do I report a search so someone else can reproduce it?
- Should papers found by citation chasing be counted in my review flow?
- How do I stop my own citations from breaking my review's criteria?