Survey Design and Response Rates in Practice
Two examined research works, two questionnaires, two very different outcomes at the letterbox. What follows is what those researchers actually decided, what they got back, and how they defended it — which is not the same as what you should do.
Two instruments, and what their design decisions actually were
The supplied teaching source names questionnaires as one of thirteen data collection methods and defines none of them. The examined works fill part of that hole by showing instruments that were built, distributed and defended in front of examiners. That is evidence about practice, not instruction — read it as a set of decisions someone had to make, each with a stated reason.
Exemplar B is a doctoral thesis using a sequential explanatory mixed-methods design, where the survey ran first and interviews followed to interpret it. Exemplar C is a portfolio-format professional doctorate in which the questionnaire is the empirical core of one of its three linked papers. The two instruments are set out side by side below.
TWO QUESTIONNAIRE INSTRUMENTS, AS DESCRIBED IN THE EXAMINED WORKS
| Design decision | Exemplar B (doctoral thesis) | Exemplar C (portfolio thesis, paper 2) |
|---|---|---|
| Delivery | Anonymous internet questionnaire | Distributed to a named sampling frame |
| Structure | Nine sections | Three sections: factual and demographic; ranking of activities; three question sets on measurement criteria, contribution of standard tools, and other tools used |
| Item types | Scaled items throughout | Deliberate mix of open and closed items |
| Scale | 7-point Likert | Likert 1–5 in section 2; Likert 0–5 in one part of section 3, where 0 = not used |
| Population approached | Senior R&D decision-makers across three world regions | Members drawn from the registers of four professional institutes, over 5,000 members in total |
| Constraint on the frame | Not stated in the material available | Confidentiality restrictions limited the coverage the researcher could obtain |
| Instrument documented? | Yes — reproduced as 30 pages of page-by-page screenshots in an appendix | Yes — both questionnaires reproduced in full, occupying 39 of the thesis's 146 pages |
Both rows of figures are findings of the individual studies. Neither is a standard for instrument length, section count or frame size.
Notice what the last row costs: thirty pages of one document and roughly a quarter of the other, spent reproducing instruments. That is the price of a survey strand someone else could repeat — see Appendices and Research Instruments.
Choosing the scale, and where to put the zero
The teaching source says only that a 5 or 7 point scale is often used to capture a response, and that data represented in ordered categories is ordinal data whose order may be reflected in the analysis. It offers no rule for choosing between five and seven points. Both examined works chose, and one of them did something more interesting than choosing.
The problem the zero point solves is a real one and it is easy to walk into. If you ask how much a project management tool contributed on a scale of one to five, a respondent who has never used that tool has two bad options: leave it blank, or pick 1. Both are wrong in different directions.
A blank is ambiguous — non-use, oversight or refusal. A 1 is worse, because it merges 'used it, found it almost useless' with 'never used it', and drags the item mean down in a way that reads as a judgement about the tool rather than a statement about its adoption. An explicit zero splits those two populations at collection time, while the respondent still knows which one they are in.
The scale decision also constrains the analysis you can honestly run afterwards. The teaching source treats ordered-category data as a form of qualitative data while acknowledging the ordering may be reflected in analysis; that framing and its tensions are unpacked in Attribute, Ordinal and Numerical Data.
Two questions that were not about the topic
The instrument reproduced in Exemplar B's appendix contains two items that carry no subject-matter content at all: a consent question and a language screening question. Both are worth copying, and for different reasons.
- The consent question puts agreement inside the instrument, attached to the response, rather than in a separate form to be matched up later. It sits alongside a participant invitation letter carrying the voluntary-participation statement, data retention and destruction terms, confidentiality wording and an ethics committee contact.
- The language screening question filters respondents on their ability to engage with the instrument as written. In a survey issued across three world regions, that is a data quality control, not a courtesy — a respondent who misreads a scaled item still produces a number, and that number is indistinguishable from a considered one.
The response funnel, in numbers that belong to one study each
Response rate is usually reported as a single percentage, which hides the shape of the loss. Both examined works allow the funnel to be reconstructed, and the two shapes are quite different.
THE TWO RESPONSE FUNNELS AS REPORTED
| Stage | Exemplar B | Exemplar C |
|---|---|---|
| Frame or list available | Senior R&D decision-makers across three regions | Over 5,000 institute members, coverage limited by confidentiality restrictions |
| Approached | 525 invitations | 150 questionnaires distributed |
| Responded | 92 responses | 31 returned |
| Excluded | 21 incomplete and rejected | None reported |
| Analysed | 71 valid responses | 31, reported as 'just over 20%' |
| Respondent profile reported | 26.2% C-level, 54.8% director-level, 19.0% senior manager | Captured in the factual and demographic section of the instrument |
Findings of two individual studies. These are not typical rates, target rates or acceptable rates for any population.
Two things follow from reading the funnel rather than the headline. The exclusion stage is itself a design decision — Exemplar B defined incompleteness as disqualifying and reported the count, so a reader can see what was discarded. And the frame stage is where Exemplar C's real constraint sits: a nominal population of over 5,000 became 150 approaches because of confidentiality restrictions, long before any respondent decided whether to reply.
Two ways of defending sample adequacy, and only one is a citation
Both researchers knew their samples would be challenged, and both wrote a defence. The defences are not equivalent, and the difference between them is the single most useful thing on this page.
Exemplar B — defended by citation
- Adequacy argued by citing two published participants-per-predictor rules of thumb.
- The rule is external to the study, so a reader can go and check it.
- It is tied to the planned analysis — the number of predictors in a regression model determines how many participants the rule requires.
- The thesis still states low response rate and small dataset as limitations, and confines the findings to the 71 companies studied rather than the whole industry.
Exemplar C — defended by assertion
- The researcher concedes a higher return would have been preferable.
- Adequacy is then asserted: just over 20% is 'considered viable for analysis'.
- No standard, threshold or published rule is cited for that figure.
- This is the researcher's own judgement recorded in the text. It is not a benchmark and must never be repeated as one.
- Limitations are stated openly in a subsection: single industry, single geography, small sample, self-selected volunteers, low response rate.
The honest form of the assertion defence is the one Exemplar C used: concede the shortfall, state the judgement as a judgement, and carry it into the limitations. What is not defensible is reporting a percentage and moving on, which invites the reader to assume a standard exists — see Stating Limitations and Contribution.
What each sample then allowed them to claim
Sample size is not an abstract virtue. It determines which analytical moves are available, and both works show the connection working in opposite directions.
SAMPLE, ANALYSIS AND CLAIM ACROSS THE TWO SURVEY STRANDS
| Exemplar B | Exemplar C | |
|---|---|---|
| Analysed responses | 71 valid | 31 returned |
| Analysis run | Descriptive statistics; one-way ANOVA with post-hoc tests at p = 0.05 across seven specified relationships; multiple regression with normal P-P plots and residual scatterplots reported | Arithmetic mean 'to indicate the tendency and provide a general picture', with tables also reporting median, mode and standard deviation |
| Inferential testing | Yes | None. No significance testing and no named software |
| Robustness move | The key construct operationalised through five different proxies and tested against each | Where two means were very close, the researcher noted they were too close to conclude anything, and contrasted that with a result where the gap was treated as meaningful |
| Scope of the claim | Findings hold for the 71 companies studied, not necessarily the whole industry | Contribution framed as widening circles: locally, then to the same profession elsewhere, then to other industries |
Exemplar C's contribution claim sits in visible tension with its own limitations subsection a page earlier. The generalisation to other industries is asserted rather than demonstrated, from 31 returns in a single industry and a single geography. That tension is worth studying precisely because the same document contains both halves of it.
Exemplar C's hedging on close means is the habit worth copying. Descriptive statistics can carry an argument if the researcher says out loud when a gap is too small to mean anything, and separately when it is not.
Designing your own survey strand
The sequence below is assembled from decisions visible in the two examined works and from the teaching source's own questions about data. The source does not present it as a procedure.
From research question to distributed instrument
Name the analysis before you write the items
Exemplar B's sample defence works only because the analysis was known in advance — a participants-per-predictor rule means nothing until you know how many predictors you have.
Build the frame, then measure how much of it you can actually reach
Exemplar C's frame was over 5,000 members and its reach was 150. Record both numbers and the reason for the difference. The gap between the two is a limitation whether or not you report it.
Choose the scale and label every point in words
Five points or seven; and if any item admits of genuine non-use, decide whether zero belongs on the scale. Print the labels on the instrument so respondents and readers interpret the same scale.
Add the items that are not about your topic
Consent, and any screening question your population requires. Put them at the front, where they filter before effort is spent.
Define disqualification in advance
Decide what makes a response unusable — incompleteness, failed screening, straight-lining — and write the rule down before you see the data. Then report the count removed, as Exemplar B reported 21.
Pilot the open items hardest
Exemplar C mixed open and closed items deliberately, and the open ones surfaced 55 additional tools beyond those in the literature. Open items are where a survey earns its keep and where ambiguity does most damage.
Reproduce the instrument in an appendix
Both works did. Without it, no reader can tell whether an item measured what your results claim.
Pre-distribution check
- The analysis you intend to run is named, and your sample target is derived from it
- The sampling frame is documented, with the reason for any gap between the frame and the people you can actually approach
- Every scale point is labelled in words on the instrument itself
- Items that a respondent could have no experience of have an explicit non-use or not-applicable option
- A consent item and any necessary screening item sit at the front of the instrument
- The rule for rejecting a response is written down before distribution
- You can state, in one sentence, the population your findings will apply to — and it is not larger than your frame
Where the survey is one strand of a larger design, the sequencing matters as much as the instrument; see Mixed Methods Design: A Worked Example and, for the strand that follows the survey in Exemplar B, Interview Design in Project Research.
What to carry forward
- Report the funnel, not just the percentage: frame, approached, responded, excluded, analysed.
- Decide before distribution whether non-use is a data point or a missing value, and put it on the scale or outside it accordingly.
- Consent and screening items belong inside the instrument, at the front, where they cost the least.
- Sample adequacy defended by a citation is checkable; adequacy defended by assertion must be labelled as the researcher's judgement and carried into the limitations.
- 'Just over 20% is viable for analysis' was one researcher's judgement about one study. It is not a threshold and does not travel.
- The analysis you plan determines the sample you need — so name the analysis first.
- Reproduce the instrument in an appendix, or nobody can check what your numbers measured.
Frequently asked questions
What response rate should I aim for?
Neither the supplied teaching source nor the examined works establishes one. One examined study returned 71 valid responses from 525 invitations and called the response rate a limitation; another returned 31 from 150 and the researcher judged that just over 20% was viable for the analysis he intended. Both are findings of single studies, not benchmarks — set your own target from the analysis you plan to run and cite whatever rule you use.
Should I use a five-point or a seven-point scale?
The teaching source notes that 5 or 7 point scales are commonly used and gives no rule for choosing. One examined work used seven points across nine sections; another used five points in one section and a 0–5 scale elsewhere. The choice should follow from how finely respondents can genuinely discriminate and from the analysis you intend, and you should cite a methods text for it rather than the practice of these studies.
Why would you put a zero on a Likert scale?
To separate non-use from low use. One examined instrument used 0 = not used in a section asking how much various tools contributed, so that a respondent who had never used a tool did not have to choose between leaving the item blank and scoring it 1. Without that option, the item mean silently mixes people who found a tool useless with people who never touched it.
Do I need to include my questionnaire in an appendix?
Both examined works did, at real cost — 30 pages of instrument screenshots in one, and 39 of 146 total pages of instruments in the other. Without the instrument, a reader cannot verify that an item measured what the results claim, so reproduce it unless there is a confidentiality reason not to, and say so if there is.
How should I report the demographics of a small sample?
Report counts alongside percentages. One examined thesis reports its respondent mix to one decimal place on a base of 71 valid responses, where a single respondent shifts each figure by roughly 1.4 percentage points. Giving the raw count lets the reader calibrate the precision of the claim for themselves.
My sampling frame is much bigger than the number of people I can actually contact. Is that a problem?
It is a limitation you must record, not a fault that invalidates the study. One examined work had access to registers covering over 5,000 members but distributed 150 questionnaires because confidentiality restrictions limited coverage. State the frame, state the reach, state the reason for the difference, and keep your claims inside the reach.
References and source attribution
- Bryman, A. 2016, Social Research Methods, 5th ed., Oxford University Press, Oxford.
- O'Leary, Z. 2017, The Essential Guide to Doing Your Research Project, 3rd ed., Sage Publications, London.
- Naoum, S. G. 2013, Dissertation Research & Writing for Construction Students, 3rd ed., Routledge.
- Veal, A. J. 2005, Business Research Methods: A Managerial Approach, Longman.
- The examined exemplar set supplied with this library: five research works comprising two doctoral theses, a portfolio-format professional doctorate, a journal paper and a conference paper. Authors, institutions, employers and publishers are not named, per the terms under which this library was written.
- The supplied teaching source: consolidated week 2 to week 4 teaching notes and slide material on research methods, used here as the basis for all statements attributed to "the teaching source".
Suggested questions for Ask KEVOS
- Help me build a response funnel table for my own survey, from sampling frame to analysed responses.
- Draft a sample adequacy paragraph for my methodology chapter that cites a rule rather than asserting a rate.
- Which of my questionnaire items need a 'not used' or 'not applicable' option?
- Write the consent and screening items for a survey issued to project managers in three countries.
- What limitations should I state if my response rate is low and my sample is from one industry?