Questionnaires and Response Rates
Two slides carry everything the supplied teaching source has to say about who returns a questionnaire and who does not. One of them contains a percentage that has travelled further than it deserves, and this page sets out exactly how far you can carry it.
What the two questionnaire slides actually say
Questionnaires arrive in this material immediately after the claim that they "are not costly and the same tool can be sent to a wide range of people", which sits at the end of the survey construction slide covered on Survey Research and Survey Construction. The next two slides then spend their whole length explaining why the returns may be worthless. The sequence is worth noticing before you read either.
Four claims sit in that slide: an example of who replies and who does not; a judgement that postal and internet questionnaires are the least satisfactory method; a percentage; and a one-line mention of automated phone surveys, which is never returned to and to which no advantage, drawback, cost or ethical consideration is attached anywhere in the material.
The 15% figure, and why it cannot work as a benchmark
This is the only response-rate figure in the supplied material, and the only number attached to the performance of any data collection method in the whole week. Because it is the only one, it attracts weight it cannot bear. Look at what is actually attached to it.
WHAT A USABLE BENCHMARK CARRIES, AND WHAT THIS FIGURE CARRIES
| What you would need to use a figure as a benchmark | What this material supplies |
|---|---|
| The study or dataset it comes from | Nothing. The figure is asserted with no attribution of any kind |
| The population surveyed | Nothing |
| The year, and the country | Nothing |
| How the sample was drawn and how large it was | Nothing |
| How a "return" was counted — delivered, opened, started, completed, usable | Nothing |
| The subject matter and the sponsor of the survey | Nothing |
| The administration mode | "postal surveys" — the only detail given |
| A stated confidence in the number | "may be a good response" — hedged in the sentence that states it |
Every row but the last two is empty. A figure with one attribute and no provenance is a remark, not a measurement.
There is a second reason the figure is unusable even if it were attributed. The same slide says the returns are "often of little value as a sample, since they are frequently biased in one direction or another". A response rate on its own says nothing about whether the people who replied resemble the people who did not. A high rate of a biased return is still biased, and the slide's own worked example shows why.
The worked example of who replies
The consumer satisfaction example is the material's clearest piece of reasoning about questionnaires, and it carries no numbers at all — no number sent, none returned, none complaining. It is offered explicitly as an example, and it is worth reading as a mechanism rather than as a story about consumers.
The mechanism the example describes
A questionnaire goes out on satisfaction
The instrument is neutral. Nothing about the questions causes the problem that follows.
People with complaints reply
The source's wording: they are "only too pleased at the chance to air them". Having something to say is the motive to reply.
Satisfied people do not
"Satisfied consumers will probably not bother to reply." Nothing has been done to them and nothing has been asked of them that they want to answer.
The returns are a sample of the motivated, not of the population
The set of returns is now selected on the thing being measured. The source's conclusion is that any inference from it is false or distorted.
Notice what this mechanism is not. It is not a sampling problem in the sense of who you approached; it is a problem of who answered. You can select your recipients perfectly and still receive a distorted return, which is precisely where the material argues with itself.
Where the notes and the slides disagree about bias
The week's study notes
- "Ideally … you'd randomly pick people from the population of people of interest to be in the study."
- "By randomly picking people, you know that there is no way any of the results could be attributed to the biases in your selection of research participants."
- Immediately withdraws random selection as "not always possible, even if your population was quite small", and offers no alternative procedure.
- "How you conduct your study is entirely up to you, so long as you do it convincingly, and within ethical considerations."
The week's slide deck
- Returned questionnaires "are often of little value as a sample, since they are frequently biased in one direction or another".
- Makes it a separate condition of use that a researcher establish that failure to reply "is in no way connected with any bias".
- Rules postal questionnaires out entirely unless one of three conditions applies.
- Names postal and internet questionnaires as "the least satisfactory method".
When a postal questionnaire may be used
The second slide is a rule, not a discussion, and it is one of very few prohibitions in the week. Quoted exactly: "Postal questionnaires are not recommended unless one of the following conditions applies:". Three conditions follow, and the rule is disjunctive — one is enough.
The three conditions, as printed
Reporting a return you cannot benchmark
The source stops at the prohibition. It gives no procedure for improving a return, no reminder protocol, no incentive guidance and no method for describing non-response in a write-up. The following is this page's construction and not the source's; it is offered because the reader still has to report something.
What to record so that your return rate can be read by someone else
- The denominator: how many people the instrument reached, and how you know it reached them
- The numerator, defined: started, completed, or usable after screening — state which, because they are three different numbers
- The distribution mode, since the one figure in the material is stated for postal surveys only
- The dates the instrument was open, and any reminders sent, with their dates
- Anything you know about the people who did not reply — role, site, discipline, whatever your list already held
- Any comparison you can make between respondents and the full list on those known attributes
- The direction you think a bias would run, and the reasoning, stated as reasoning rather than as a finding
- A plain statement in your limitations that no benchmark for the rate is available to you from this material
What the week's own audit records about this topic
One thing belongs on this page in the reader's interest, stated once. The week has a single stated learning objective — to evaluate methods of data collection, including sampling, interviewing structures and techniques, questionnaires, observations, focus groups and surveys. The extraction audit records the verb as undelivered: the week performs no evaluation, because no two methods are ever compared with each other on any criterion.
The material for that evaluation, in total, is three items: an advantages and disadvantages table for one distinction, covered on Structured and Unstructured Questions; the 15% figure on this page; and the prohibition with three exceptions on this page. That is the whole evidential basis the week offers for preferring one collection method to another, which is why the questionnaire content should be read as description rather than as recommendation.
What to carry forward
- The material's judgement is that postal and internet questionnaires are the least satisfactory method because relatively few come back and the returns are frequently biased.
- The 15% return figure is the only performance number in the week: unattributed, double-hedged, tied to postal surveys and to nothing else. Never treat it as a benchmark, a target or a prediction.
- Internet methods are said to "usually improve the percentage" with no figure attached to the improvement.
- The consumer satisfaction example is a mechanism worth keeping: replies come from people with a motive to reply, which selects the return on the thing being measured.
- Postal questionnaires are permitted only under three conditions — legal obligation, interviewing all non-responders, or establishing an absence of bias the same deck says is normally present.
- The study notes and the slides disagree about whether random selection removes bias. Both are reproduced; neither is adopted.
- Nothing in the material covers internet questionnaires as a distinct mode, though that is the one most project researchers will use.
Frequently asked questions
Is a 15% response rate good?
The supplied material says a return of 15% on postal surveys "may be" a good response, and attaches no study, population, year, country or definition of "return" to that statement. It is an unattributable remark and it cannot tell you whether your own rate is good. If you need a defensible expectation, take it from published survey work in your own field.
Can I cite the 15% figure in my proposal or methodology chapter?
No. Citing it would attribute to the material a benchmark it does not establish and hedges twice in the sentence that states it. If you must mention it, describe it as an unattributed rule of thumb appearing in teaching material, and do not use it to justify a sample size, a target or a judgement about your own return.
When does this material permit a postal questionnaire?
Only if one of three conditions applies: completion is a legal obligation, such as a census; all non-responders are subsequently interviewed; or an appropriate sample of non-responders is interviewed and it is established that failure to reply is in no way connected with any bias. No size is given for that sample and no procedure is given for establishing the absence of bias.
Do those three conditions apply to an online questionnaire?
The material does not say. The previous slide groups postal and internet questionnaires together as the least satisfactory method, but the three conditions are stated for postal questionnaires only and are never extended. Given that most project researchers distribute electronically, the silence is a real gap rather than a technicality.
Does random selection protect me from response bias?
The week's two documents disagree. The study notes say that by randomly picking people there is no way results could be attributed to biases in your selection of participants; the slides say returned questionnaires are frequently biased in one direction or another and make the absence of that bias a separate condition to be established. The material never reconciles the two positions.
How should I report non-response in my write-up?
The source gives no procedure, so this is a matter for your own judgement. Record the denominator, define what counts as a response, note the mode and dates, and compare respondents with your full distribution list on whatever attributes you already hold. Report the comparison and the direction any bias would run, and state plainly that no benchmark for the rate is available from this material.
References and source attribution
- Bryman, A. 2016, Social Research Methods, 5th ed., Oxford University Press, Oxford.
- Veal, A. J. 2005, Business Research Methods: A Managerial Approach, Longman.
- O'Leary, Z. 2017, The Essential Guide to Doing Your Research Project, 3rd ed., Sage Publications, London.
- Trochim, W. M. K. 2006, Research Methods Knowledge Base.
- The supplied teaching source: the week 10 slide deck on collecting data, slides on questionnaires and response rates, read together with both copies of the week 10 study notes.
Suggested questions for Ask KEVOS
- Draft the non-response paragraph for my limitations section, given I cannot benchmark my return rate.
- What should I record about my distribution list before I send my questionnaire out?
- Is my questionnaire likely to be returned mainly by people with a motive to reply, and how would I tell?
- Which of the three postal-questionnaire conditions could a project-scale study realistically meet?
- Help me define what will count as a completed response before any responses arrive.