Citation Health in Examined Work
Twenty in-text sources across seven examined works cannot be reached from the documents that cite them, and in six of the seven at least one of them carries a load-bearing claim. The headline metric everyone reaches for measures something else entirely.
Two different failures, counted separately
Reference-list problems come in two shapes that behave nothing alike, and reporting them as one number is what makes citation health hard to reason about. The seven works measured here are mapped at Eight Examined Theses Compared.
An uncited entry
- A work sits in the reference list and is never cited in the body.
- Cost to the reader: nothing. They can still find everything the document points them to.
- It is a tidiness fault, and in some designs it is not a fault at all.
- Detection is mechanical: match every entry against the body.
An unresolvable citation
- A source is cited in the text and cannot be found in the list — or is listed under a different name, year or author order.
- Cost to the reader: the source. They cannot check the claim it carries.
- In six of seven works here, at least one unresolvable citation carries a load-bearing claim.
- Detection requires reading: match every in-text citation to an entry, by surname and year.
The two are opposite failure modes and this corpus has both, in different works. Teach them apart, measure them apart, and fix them in different ways.
Seven works, measured
CITATION HEALTH ACROSS SEVEN EXAMINED WORKS
| Work | Entries | Uncited | Uncited % | In-text sources absent from the list | Further resolution defects | Words per entry |
|---|---|---|---|---|---|---|
| F | 25 (+5 in a separate bibliography) | 0 | 0% | 4 | 0 | 471 |
| G | 66 | 4 | 6.1% | 0 | 3 | 213 |
| H | 50 printed, 49 distinct | 5 | 10.2% | 3 | 10 | 216 |
| J | 42 | 3 | 7.1% | 5 | 7 | 257 |
| K | 70 | 32 | 45.7% | 2 | 15 | 151 |
| L | 23 | 4 | 17.4% | 4 | 13 | 424 |
| M | 18 | 1 | 5.6% | 2 | 13 | 609 |
| All seven | 294 | 49 | 16.7% | 20 | 61 | — |
Counts verified entry by entry for this library's record, by first-author surname and by title keyword. One further examined work outside this seven is recorded elsewhere in the library at 14 of 64 entries uncited (22%). Two of the supplied extracts differ by one on the uncited count for work H (5 against 6); the reconciled figure that feeds the batch total is used here.
The spread across one section template is 0% to 45.7% — nearly forty-six points. That alone should stop anyone quoting a single uncited percentage as a quality signal. Why the spread is that wide is the next section, and it is not that one researcher was six times less careful than another.
The metric that mismeasures a systematic review
The systematic review is the case that breaks the metric. Its search flow reconciles: 27 plus 321 records screened gives 348; 27 plus 288 excluded gives 315; 33 papers retained. Its reference list runs to 70 entries, and 32 of them are never cited anywhere in the body. The two numbers are one apart.
The wider point transfers. Before quoting any quality metric across a set of documents, check that the documents are the same kind of thing. See A Worked Systematic Literature Review for what that review reports, and why its corpus was never identified.
The failure that costs the reader the source
Twenty distinct in-text sources across the seven works cannot be reached from the documents that cite them. In six of the seven, at least one carries a claim the work depends on. For the anatomy of an entry that resolves, see Harvard Reference Anatomy.
LOAD-BEARING SOURCES THAT DO NOT RESOLVE
| Work | What is missing from the list | What the missing source carries |
|---|---|---|
| F | Four in-text citations, all of them the study's methods sources | The justification for choosing qualitative research; the two data-collection methods; the four-to-fourteen focus-group size rule, cited four times; and the thematic analysis method. Twenty-five references resolve; the four that carry the method do not |
| H | Three sources | All three carry the work's central finding: a consultancy article cited six times, a book quoted verbatim, and a source quoted twice with page numbers. The audit's own evidence cannot be reached from the document that contains it |
| J | Five distinct sources | Including the citation carrying one of the four moves of its literature-gap argument, and a professional-services report cited three times, twice with page numbers |
| K | Two sources | One carries the whole of the second half of a discussion section; the other is the conceptual-fit theory that opens a discussion sub-section, named by author and title with no year. Two out of two are load-bearing |
| L | Four sources | Including the sole support for the closing claim of one results sub-section, and a source cited under a third spelling of a surname the reference list spells a fourth way |
| M | Two sources | A body-of-knowledge standard supplying the work's most formal definition of its central term — the list carries the same issuer's later edition and no matching entry — and a source carrying two quoted claims in the literature review |
References carrying a neighbour's address
Between absence and correctness sits a category of defect that is easy to miss and cheap to prevent: an entry that exists, looks complete, and sends the reader somewhere else.
- A conference paper listed with another paper's address. One 2016 entry carries the web address of a 2007 paper by a different author. The entry resolves — to the wrong document.
- Three entries whose location is a local hard-drive file path. They resolve on one machine and nowhere else, and the path string also carries a fragment of a personal name. In the same work, one further entry prints the bare word "Link" where the address should be.
- A body address that disagrees with the list address. One work gives a regulator's web address in the body with a commercial top-level domain and in the reference list with a government one.
- A citation attached to the wrong subject. One entry, listed as a study of one country's games in one year, is cited in the body as evidence that a different country's games six years earlier used a particular methodology successfully.
- A corporate source cited by acronym and listed under its issuer's full name, so the string the reader searches for appears nowhere in the list. Two works in the corpus do this on the same source, and in one of them the full institutional name occurs zero times in the body.
Defects short of absence
Sixty-one further defects were counted across the seven works — problems that do not remove a source but make it harder to reach than it should be. They cluster in whole documents rather than scattering evenly, which points at record-keeping rather than carelessness: see Note-Taking and Bibliographic Records.
RESOLUTION DEFECTS SHORT OF ABSENCE, BY TYPE
| Defect | Count across seven works | Worst single instance |
|---|---|---|
| In-text surname misspelled against the list | 12 | One work misspells one first author's surname three different ways across five citations of the same source |
| In-text year differs from the list | 9 | A 2014 entry cited as 2013 five times and as 2014 once, in one document |
| Alphabetical order broken | 6 entries | Three surnames beginning U and V filed after the W entries, at the end of a 70-entry list |
| List entry missing its year | 5 | The single most important primary source in one work is listed without the year its own argument turns on |
| Author order reversed between text and list | 3 | The list files under a surname the reader is not looking for |
| Same author and year listed twice with no (a)/(b) | 4 pairs | Four ambiguous pairs in one list, one of them the same article printed twice |
| One source rendered under three or more spellings | 2 | One work renders one author's surname four ways — three in the text, a fourth inside the entry's own web address |
| In-text citation with no author at all | 2 | A 2017 report cited as "(et.al, 2015)" — "et al" with no surname and the wrong year |
| List entry missing its author | 2 | One entry begins with its year and is cited in the text by an initialism that appears nowhere in the list |
Counts are this library's, verified individually across the seven works. They describe these documents only.
The audit that takes an afternoon
Every defect on this page is detectable before submission with nothing beyond a word processor's find function. None of these seven documents had the check run on it, and all seven passed anyway — which is the argument for running it yourself.
Six passes over your own reference list
Every in-text citation to a list entry
Go through the body, not the list. For each citation, find the entry by surname and year. Anything that does not match on both is a defect, even if you know what it means. This pass catches the twenty absences and most of the year and surname mismatches.
Every list entry to at least one in-text citation
Now go the other way. Anything with no citation is either untidiness, drafting residue, or — if you have run a review — corpus you have not labelled. Decide which, and act accordingly: delete, or label.
Open every address
Click each one. This catches hard-drive paths, placeholder words, addresses pointing at other people's documents, and body-versus-list disagreements. It is the dullest pass and the highest yield.
Sort and scan
Sort the list alphabetically and read the first letters down the page. Six order breaks across this corpus would have been visible in thirty seconds.
Same author, same year
Search for duplicated surname-and-year pairs. Add (a) and (b) in both the text and the list, or the citation is ambiguous. One work in the corpus does this correctly; another leaves four ambiguous pairs.
Search your own text for every corporate author
If you cite an issuer by acronym and list it under its full name, the string your reader searches for appears nowhere. Add the acronym to the list entry, or cite by the listed name.
What to carry forward
- Count uncited entries and unresolvable citations separately. They are opposite failures with opposite costs, and only one of them harms your reader.
- Do not compare uncited percentages across designs. A systematic review is supposed to list a corpus it does not quote; measuring it against a narrative thesis compares two different objects.
- If your review has a retained set, label it. One line in the reference list or one appendix converts a 45.7% "defect" into a documented corpus.
- The unresolvable citation is the one to hunt. Twenty across seven works, and in six of the seven at least one carried a claim the work depended on.
- A clean list is not a complete list. The tidiest reference list in this corpus is the one missing all four of its methods sources.
Frequently asked questions
What is an acceptable proportion of uncited references?
No figure exists in the supplied material and none of the counts on this page is a benchmark. The observed range across seven examined and passed works is 0% to 45.7%, and the top of that range belongs to a systematic review whose list plausibly includes its review corpus. The useful target is not a percentage; it is that every entry has a reason to be there and every reason is visible to a reader.
Why does the standard uncited metric fail on a systematic review?
Because the metric assumes every listed entry should be cited in the body, and a review's reference list is doing a different job: it lists the corpus that was screened and retained. Quoting only some of that corpus is normal practice, not sloppiness. Reporting a review at 45.7% uncited beside a narrative thesis at 6% compares a corpus with a bibliography.
Which is worse — an uncited entry or a citation that does not resolve?
The unresolvable citation, by a wide margin. An uncited entry costs a reader nothing; they can still reach everything the document points to. An unresolvable citation costs them the source, and in six of the seven works here at least one of them carried a claim the work depended on — including, in one case, all three primary sources behind that work's central finding.
How do I label a review corpus in my reference list?
The corpus does not supply a convention, because the one work that needed one did not use it. Anything unambiguous works: a marked sub-list, an asterisk against retained entries with a note explaining it, or a separate appendix listing the retained set. What matters is that a reader can tell the corpus from the works you cited, and neither of the two counts is then misread.
What is a resolution defect short of absence?
A source that is present in both the text and the list but hard to match: a misspelled surname, a year that differs between the two, reversed author order, a missing (a)/(b) on a duplicated author-and-year pair, a corporate author cited by acronym and listed by full name. Sixty-one were counted across these seven works. None removes a source; all of them slow a reader down or send them to the wrong entry.
Do these figures say anything about theses in general?
No. They describe seven examined documents, each measured individually. The defects are visible because the reference lists were printed and could be audited entry by entry; a work that publishes less is not cleaner, only less checkable. Every count on this page belongs to the study that produced it.
References and source attribution
- Seven examined master's works in project management, supplied as student work, whose reference lists were audited entry by entry — by first-author surname and by title keyword — against every in-text citation. Researchers, supervisors, institutions, jurisdictions and employers scrubbed. Used as observed practice, not as model answers.
- The consolidated extract of four of those works, which records 183 printed entries, twelve uncited, twelve in-text sources absent from the lists, and twenty further resolution defects, together with the finding that uncited entries and unresolvable citations are opposite failure modes.
- The consolidated extract of the three works examined afterwards, which records the corrected citation metric used on this page: that the uncited-reference measure captures tidiness, drafting residue and a genre property in three different works and should not be reported as a single number across designs.
- A further examined work recorded separately in this library at 14 of 64 reference entries uncited, cited here only as a comparator outside the seven.
- The supplied teaching source: weekly study notes, slide decks and assessment activities for a master's-level research methods subject in project management, which sets no threshold for reference-list health. Author, institution and year not stated in the supplied files.
Suggested questions for Ask KEVOS
- How do I audit my reference list before submission?
- Why does a systematic review's reference list have so many uncited entries?
- How should I label the retained corpus of my literature review?
- What is the difference between an uncited entry and an unresolvable citation?
- How do I handle two works by the same author in the same year?
- How do I cite a corporate author I refer to by acronym throughout?