Designing a survey that measures what you need: objectives, questions, answers and testing (Research and Evidence for Better Decisions) | READOS

Surveys look easy. Anyone can write a list of questions, load them into an online tool and send a link to customers or staff. Within days, responses arrive, charts appear automatically and a report can be written. The ease is deceptive. Poorly designed surveys produce numbers that look precise but measure the wrong thing, reflect the wording of the questions more than people’s real views, or come from respondents who do not represent the people the business cares about. Decisions based on such surveys can be worse than decisions made without any data, because the numbers carry unearned authority.

A good survey is a measuring instrument, and like any instrument it must be designed for its purpose, calibrated and checked. That means starting with the decisions the survey will inform and the information they need, translating that information into clear questions with suitable answer options, arranging the questions so that earlier ones do not distort later ones, keeping the survey short enough that people finish it thoughtfully, and testing it with real people before sending it to everyone. It also means planning the analysis before collecting a single response, so that every question earns its place.

This article explains how to set survey objectives, write questions and answer options, avoid common biases, order and format a questionnaire, choose how to deliver it, test it, encourage responses, handle privacy and plan the analysis, with a worked example. It is part of a series on research and evidence for business decisions.

Start with decisions and information needs

Before writing questions, write down the decisions the survey will inform and the information each decision needs. For a customer survey, the decisions might include which service improvements to prioritise, whether to offer a new product and how to price it. The information needed might include satisfaction with specific aspects of service, the importance of each aspect, current purchasing behaviour and willingness to consider alternatives.

Each question should trace back to an information need. Questions that do not, the “it would be interesting to know” questions, lengthen the survey, reduce completion and dilute attention. A useful test is to sketch the table or chart each question will produce and the decision it will inform. If no decision follows, the question can go.

From ideas to measurable questions

Many things businesses want to know are abstract: satisfaction, loyalty, trust, ease of use, perceived quality. Survey design turns these concepts into specific questions that respondents can answer. “Satisfaction with delivery” might be measured through questions about on-time delivery, accuracy of orders, condition on arrival and communication about delays. Breaking abstract ideas into concrete parts makes answers more reliable and more useful, because they point to what to fix.

Writing clear questions

Good survey questions are:

Clear and specific: “How many times did you contact our support team in the last three months?” rather than “Do you contact support often?”
About one thing: avoid double-barrelled questions such as “How satisfied are you with the speed and quality of our service?”, which cannot be answered if speed and quality differ.
Neutral: avoid leading questions such as “How much did you enjoy our new, improved ordering system?”
In the respondent’s language: avoid jargon, acronyms and internal terms.
Answerable: ask about things respondents know and can remember. Recall of exact numbers fades quickly, so use recent, bounded time periods.
Free of double negatives, which confuse respondents.

Sensitive questions, such as those about income, problems or compliance, should be phrased carefully, placed later in the survey and made optional where possible.

Designing answer options

Answer options matter as much as questions:

Categories must be exhaustive and mutually exclusive: every respondent should fit one category, and only one. Age bands of 20–30 and 30–40 overlap at 30.
Rating scales should be balanced, with equal numbers of positive and negative options and clear labels, such as “very dissatisfied” to “very satisfied”.
Include “don’t know” or “not applicable” where some respondents genuinely cannot answer, so they are not forced to guess.
Keep scales consistent throughout the survey to avoid confusion.
Use open questions sparingly but deliberately, to capture reasons and suggestions in respondents’ own words.

The next article in this series, on question types and levels of measurement, explains how the type of answer affects the analysis that is possible.

Biases in responses

Even well-worded questions are subject to response biases:

Social desirability: people present themselves favourably, overstating good behaviour and understating problems.
Acquiescence: some respondents tend to agree with statements regardless of content.
Recall bias: memory is imperfect, especially for frequency and timing.
Order effects: earlier questions influence answers to later ones, and the position of options in a list affects how often they are chosen.
Hypothetical bias: people predict their own future behaviour poorly, especially purchasing.

Design can reduce these effects: anonymous responses, neutral wording, mixing positively and negatively worded statements, randomising the order of options where appropriate and asking about past behaviour rather than intentions where possible.

Ordering the questionnaire

A logical order helps respondents and reduces bias:

start with easy, relevant questions that engage interest;
group related questions together;
ask general questions before specific ones on the same topic, to avoid specific questions shaping general judgements;
place sensitive and demographic questions near the end; and
use routing or skip logic so that respondents see only questions that apply to them.

Length and completion

Every additional question reduces the share of people who start and finish the survey, and long surveys produce rushed, careless answers towards the end. Most business surveys should take only a few minutes. Showing progress, keeping questions short and removing anything not tied to a decision all help. If many topics must be covered, separate surveys to different groups, or rotating modules, may work better than one long questionnaire.

Choosing the mode

Surveys can be delivered online, by phone, on paper or in person. Online surveys are cheap and fast but reach only people with the means and willingness to respond online. Phone surveys reach people who would not respond online and allow clarification, but cost more and are more prone to social desirability effects. Paper suits some settings, such as events or workplaces without computers. In-person surveys suit short questionnaires at points of service. The best mode depends on who the respondents are and how they prefer to communicate.

Testing before launch

Testing is the most neglected step. Cognitive interviews, in which a few people from the target group answer the survey while thinking aloud, reveal questions that are misunderstood, ambiguous or impossible to answer. A pilot with a small sample checks the flow, timing, routing and data output. Testing almost always finds problems, and fixing them before launch costs far less than discovering them afterwards.

Encouraging responses

Response rates improve when people understand why the survey matters and trust how their answers will be used. A short, personal invitation from someone they know, a clear statement of purpose and length, one or two polite reminders and, sometimes, a modest incentive all help. Telling previous respondents what changed as a result of their feedback makes future participation more likely. The customer feedback and net promoter score article discusses how feedback programs maintain engagement over time.

Surveys collect personal information whenever respondents can be identified. Respondents should be told who is collecting the information, why, how it will be used and stored, whether answers are anonymous or confidential and how to contact the business. Businesses covered by Australian privacy law must meet their obligations when collecting personal information, and all businesses should collect only what they need. The privacy by design in business databases article explains how to build privacy into data collection and storage.

Surveys of staff

Staff surveys raise particular issues. Employees may worry that honest answers will be traced back to them, especially in small teams where demographic questions such as department and length of service can identify individuals. Reporting results only for groups above a minimum size, avoiding unnecessary demographic questions and using an independent tool or administrator help build trust. Staff surveys also create an expectation of action: asking people about problems and then doing nothing damages trust more than not asking. Planning how results will be shared and acted on is part of the design.

Repeating surveys over time

Surveys repeated at intervals, such as annual customer surveys or quarterly staff pulse checks, show trends only if the questions, scales, sampling and timing stay consistent. Changing a question’s wording or the scale breaks the comparison with previous results. Where changes are necessary, running old and new versions in parallel for one round, or clearly marking the break in trend charts, avoids misleading comparisons. Keeping a master version of the questionnaire with a change history protects the value of long-running surveys.

Planning the analysis in advance

Before launch, plan how each question will be analysed: which percentages, averages or comparisons will be calculated, which groups will be compared and how open-ended answers will be coded. Planning reveals questions that cannot support the intended analysis, such as a rating scale too coarse to show differences or a category scheme that does not match the business’s customer segments. It also guards against searching through results afterwards for interesting patterns that are really chance.

Checking data quality

When responses arrive, check for speeders who answered implausibly quickly, straight-lining where every answer is identical, contradictory answers, duplicates and incomplete responses. Decide in advance how such cases will be treated, so that the decision is not influenced by what they say.

A worked example

This is an illustrative example. The business and figures are hypothetical.

An industrial services company wants to improve customer retention. Its first draft survey asked, “How satisfied are you with our fast and friendly service?” on a scale of 1 to 10, followed by twenty other questions added by different managers.

The operations manager rewrote the survey around three decisions: which service aspects to improve first, whether to introduce a priority response plan and how to communicate during jobs. Each question was traced to one of these decisions, cutting the survey to twelve questions. The leading, double-barrelled first question became four separate questions about response time, technician courtesy, quality of work and communication, each with a balanced five-point scale and a “not applicable” option. A question about interest in a priority response plan described the plan’s features and price range and asked about past experiences of urgent breakdowns, rather than relying only on stated intentions.

Five customers completed the survey while talking through their thinking. Two misread “response time” as the time to provide a quote, so the question was clarified. The survey went to 300 customers with a personal email from their account manager and one reminder, and achieved a 38% response rate. Results showed that communication during jobs, not response time, was the weakest area, and the company changed its job-update process before investing in faster response.

How this applies to a small Australian business

Trace every question to a decision and remove the rest.
Break abstract ideas into concrete questions.
Write neutral, specific, single-topic questions with balanced answer options.
Keep surveys short and use skip logic.
Test with a few real respondents before launching.
Plan the analysis before collecting data.
Tell respondents how their information will be used and meet privacy obligations.
Report back on what changed as a result.

Common mistakes

Leading and double-barrelled questions.
Overlapping or incomplete answer categories.
Long surveys with questions nobody will use.
Relying on stated intentions to forecast purchases.
Skipping testing to save time.
Analysing without a plan and finding chance patterns.
Changing question wording between rounds of a repeated survey without noting the break.
Ignoring non-response, assuming respondents represent everyone.

Frequently asked questions

How long should a business survey be? As short as the decisions allow. For most customer surveys, a few minutes is a sensible target.

Should rating scales have a midpoint? Usually, yes. A neutral midpoint lets people with genuinely neutral views answer honestly. Forcing a choice can be justified for specific purposes, but it should be a deliberate decision.

Are online survey tools’ default templates good enough? They are a useful starting point for formatting, but default questions rarely match a specific business’s decisions and should be rewritten.

How many open-ended questions should a survey include? One or two well-placed open questions, such as asking for the main reason behind a rating, add valuable context without making analysis unmanageable.

Questions to ask

Which decision does each question in our survey inform?
Have we tested the survey with people from the target group?
Are our answer options balanced, exhaustive and non-overlapping?
How will we analyse each question, and have we written that down?
How will we protect anonymity in small groups of staff or customers?
How will we tell respondents what we did with their feedback?

Bringing it together

A survey is a measuring instrument. Start from decisions and information needs, turn abstract ideas into concrete questions, write neutral and specific questions with balanced and complete answer options, order them carefully, keep the survey short, choose a suitable mode, test it with real respondents, plan the analysis in advance, check data quality and respect respondents’ privacy. Designed this way, a modest survey can produce evidence that genuinely improves decisions. The market research on a small-business budget article shows where surveys fit among other low-cost research methods.

Source: KEVOS editorial notes. Examples and figures in this article are illustrations. This article is general information and does not constitute legal advice on privacy obligations.