Bibliometric Trend Analysis as a Research Method
Coding a whole journal against an external standard to chart a field's research trends — the design, the bias-control logic, and the exact point where the exhibit outruns the evidence.
The design
The examined conference paper calls its method "desktop literature research" followed by a comparative analysis mapping research trends against socioeconomic events. In current terminology this is a keyword-and-abstract-based bibliometric content analysis; the paper does not use that label.
THE DESIGN AS STATED
| Element | What was done |
|---|---|
| Corpus | All research articles in one leading journal, 1983–2016 — 2,015 articles, the whole population of that outlet for the window |
| Unit of analysis | The article |
| Coding frame | The ten knowledge areas of an international project management standard: integration, scope, time, cost, quality, risk, communication, human resource, procurement, stakeholder |
| Coding basis | Keyword and abstract review against those ten areas |
| Second data stream | Socioeconomic and technological events retrieved from research databases, plus reports from an international body covering 1980–2016, matched to trends by keyword search |
| Country attribution | Nationality of the first author only |
| Analysis | Frequency counts by year and knowledge area; a country-count table; a collaboration matrix; visual juxtaposition of counts against dated events |
All counts and dates are that study's own. The ten knowledge areas come from an external published standard, not from the study.
Why one journal, and how it was defended
Restricting a field-wide trend study to a single outlet is the design's most obvious vulnerability, and the paper defends it at unusual length with four separate justifications.
- The outlet's standing in the field.
- That it is written in English, treated as the de facto standard for the field despite prestigious journals existing in other languages.
- That its scope aligns with the study's objective, because prior global-trend reviews have appeared in it.
- A cited methodological recommendation that investigating one specific reliable journal secures consistency of research topics and yields more meaningful trends than surveying a wider domain.
The bias-control device worth copying
The authors note that one of their ten categories shows no consecutive trend because keyword ambiguity makes it hard to classify — a study may be keyword-tagged to one area while its actual purpose belongs to another. They surface the problem themselves and recommend reading further into the article when ambiguity arises. Admitting that a coding frame under-captures one of its own categories is a candid move and a good model.
What the study reported
The exhibits, and the one that overreaches
HOW THE ARGUMENT IS CARRIED
| Exhibit | What it does | Verdict |
|---|---|---|
| Line chart of annual volume | Establishes growth over the window | Sound |
| Country table, 70 rows | Establishes concentration | Sound, subject to the first-author caveat |
| Collaboration matrix with a 'Note' column grouping rows by pattern | Tells the reader how to read the grouping | A genuinely well-designed exhibit worth copying |
| Small multiples — ten bar charts, one per knowledge area, each with its own y-scale | Lets the reader compare the shapes of trends rather than their magnitudes | Excellent technique; the independent y-scales are the point |
| Three-page table: rows are years, columns are the ten knowledge areas plus a wide 'Socioeconomic Events' column | Places counts and world events in the same row | This is where the exhibit outruns the evidence |
Observations about one paper's exhibits.
The paper also explains rises in a knowledge area by naming a plausible contemporaneous event. Confirming instances are supplied; disconfirming ones are not sought. That asymmetry is worth watching for in any trend argument.
What the design leaves out
When this design is the right choice
Fit test
How other examined works handled the same reporting obligations is set out in Stating limitations and contribution.
What to carry forward
- Coding against an external published standard is a cheap, strong bias-control device. Adopt it where a suitable frame exists.
- Small multiples with independent y-scales let a reader compare trend shapes rather than magnitudes. An excellent exhibit.
- A table placing two series in the same row is co-occurrence, not correlation, however convincing the layout.
- Report the search string, inclusion rules and an inter-coder check. This design reported none of them.
- Acknowledge your corpus boundary as a limitation, not only as a justification for the choice you made.
Frequently asked questions
Is analysing one journal defensible?
The examined paper defends it four ways, but only one of those is a methodological argument for narrowing at all — the other three argue which journal to choose. If you narrow a corpus, make both arguments separately and list the boundary in your limitations.
Why code against an external standard?
Because author-generated categories are always open to the objection that they were shaped by what the researcher expected to find. A published standard's categories existed before the analysis and can be inspected independently.
What is wrong with the events table?
Nothing, as an exhibit. The problem is inferential: placing counts and events on the same row invites the reader to see a correlation, which the paper then asserts while conceding it never quantified one.
What is an inter-coder reliability check?
A second person independently codes a subset of the material and the agreement rate is reported. The examined paper does not report one, which for a single-coder study of over two thousand articles is a substantial omission.
Why attribute a paper to the first author's country only?
It is simpler, and the examined study did it. The cost is that multi-country collaborations count once, which systematically understates the collaboration the paper then analyses. Attribute to all countries, or state the choice as a limitation.
References and source attribution
- Five examined research works supplied as exemplars: two doctoral theses (1999, 2016), a doctoral portfolio thesis (2004), a peer-reviewed journal paper (2014) and a conference paper (2017/2018). Structural observations only; chapter bodies were not reproduced.
- Supplied teaching source, Weeks 2-4: Introduction to Research Methods, Developing a Research Topic, Reviewing the Literature.
- Bryman, A. 2016, Social Research Methods, 5th ed., Oxford University Press, Oxford.
- O'Leary, Z. 2017, The Essential Guide to Doing Your Research Project, 3rd ed., Sage Publications, London.
Suggested questions for Ask KEVOS
- Design a bibliometric study of a field I name.
- What published standard could I use as a coding frame for my topic?
- How do I report inter-coder reliability?
- What is the difference between co-occurrence and correlation in a trend study?
- Write the inclusion and exclusion criteria for a literature corpus.