There is a category of form problem that never appears in any analytics. The form completes, the data arrives, everything looks healthy, and the answers are subtly wrong. Not falsified, exactly. Shaped by how the question was asked.
This is well documented in survey research and almost entirely ignored in ordinary business forms, which is unfortunate, because the same effects apply and the consequences are more immediate. A satisfaction score inflated by the wording of the question leads to decisions made on a number that was never real.
The good news is that the main causes are few and the fixes are matters of wording rather than of design.
People answer the question they think you want
The strongest effect is the simplest. If a question implies a preferred answer, a significant share of people will give it, not out of dishonesty but because agreeing is the path of least resistance with a stranger.
How much did you enjoy the service is not the same question as how was the service. The first has assumed enjoyment and invited a number on a scale that starts from it. Asked the second way, the same experience produces measurably lower and more useful answers.
The test is to read each question and ask whether it presumes anything. Any question containing a positive word about your own offering is presuming, and the fix is to remove the word rather than to balance it with a negative one.
How much did you enjoy it has already assumed the answer. How was it has not.
Scales are not neutral
Rating scales carry assumptions in their structure, and the most common one is being unbalanced. A scale running from excellent to acceptable has no room for a bad experience, so bad experiences get recorded as acceptable and the average looks fine.
An honest scale has as many negative points as positive ones and a genuine middle. It should also be labelled at every point rather than only at the ends, because people interpret an unlabelled four out of five very differently from each other.
The length matters less than people argue about. Five points is fine, seven is fine, ten is fine if you label them. What is not fine is four, which forces a side by removing the middle, and is sometimes chosen for exactly that reason.
The middle option earns its place
There is a persistent belief that removing the neutral option produces more decisive data. It produces more decisive-looking data, which is not the same thing.
People who genuinely have no opinion, or for whom the question does not apply, will pick something when forced, and what they pick is close to random. That noise is then indistinguishable from real signal, and it moves averages in ways nobody can trace.
Keep the middle, and add a not applicable where the question might genuinely not apply. Both reduce the volume of answers and improve the quality of the ones you get, which is the trade worth making every time.
Sensitive questions need a reason
Any question about money, age, health or anything somebody might feel judged about gets a lower response rate and a less accurate one, and both improve substantially with one sentence of context.
We ask about budget so we can suggest options in range rather than wasting your time. That converts a question that felt like assessment into one that is obviously in the respondent's interest, and the effect on answer quality is larger than any wording change to the question itself.
Offering ranges rather than exact figures helps for the same reason. A range is a category rather than a disclosure, and people answer it more readily and more accurately than a box asking for a number.
Do not ask two things at once
Double-barrelled questions are everywhere and produce answers nobody can interpret. How satisfied were you with the speed and quality of the work is two questions, and somebody who thought it was fast and sloppy has no honest answer.
What they do is average, or pick the one they felt more strongly about, and you cannot tell which from the data. The average of a five and a one looks identical to two threes, and the difference between those two situations is the whole point of asking.
The fix is mechanical. Look for the word and in any question and check whether it is joining two things you would act on separately. If it is, split the question, even though that makes the form one item longer.
Order affects answers
Questions are read in sequence and each one colours the next. Ask about problems first and the subsequent satisfaction score falls. Ask about the best part first and it rises. Neither order is wrong, and using different orders for different rounds makes the results incomparable.
The practical guidance is to put general questions before specific ones, because a specific question primes the general one that follows it. How was your experience overall, asked after four questions about delivery delays, is really a question about delivery delays.
And once you have chosen an order, keep it. Consistency matters more than optimality when you are comparing this quarter with last.
Anonymity changes what people write
For feedback in particular, whether the answer is attributable changes it substantially, and people are better at working out whether they are identifiable than form designers assume.
A feedback form that asks for a name, or that is sent to a list of six customers, is not anonymous regardless of what it says, and the answers will be softened accordingly. If you want candid feedback, either make it genuinely anonymous and say how, or accept that you are getting the attributable version and interpret it that way.
The dishonest version, which is promising anonymity while collecting enough context to identify somebody, is worse than either. People notice, and it costs you the next round as well as this one.
Required fields produce fiction
A required question that somebody cannot or will not answer does not produce a missing value. It produces a made-up one, and a made-up value is worse than a blank because nothing marks it as unreliable.
This is where the postcode fields full of the same four characters come from, and the phone numbers that are seven ones, and the company names reading N slash A. Every one of those is a person who wanted to continue and had no honest option.
Make a field required only when you would genuinely rather lose the submission than lose the answer. For most fields on most forms that is not true, and the required marker is there because it was the default.
Small wording changes worth making
These are the ones that come up most often and cost nothing to fix.
- Remove any positive adjective about your own service from the question text.
- Label every point on a scale, not just the ends.
- Split any question containing and where both halves matter.
- Add a short reason to any question about money, age or health.
- Offer not applicable wherever the question might not apply.
- Replace internal category names with the words a customer would use.
- Check that no option list forces somebody into a category that is wrong for them.
Read the answers you get as evidence about the questions
The data itself tells you where the wording is failing, if you look at the distribution rather than the average.
An answer everybody gives is a question that is not distinguishing anything. A scale where nobody uses the bottom two points is usually an unbalanced scale rather than a universally good service. A field with a high blank rate is a question people are declining, which is information about the question.
Reviewing distributions once a quarter catches most of this, and it takes a few minutes. It is also the only way to notice that a question has stopped working, which happens gradually as the business changes around it.
Option lists are questions too
A great deal of answer distortion comes not from the question but from the options underneath it, and those get much less scrutiny because they look like data rather than like writing.
The most common problem is a list that does not contain the honest answer. How did you hear about us, with six marketing channels and no option for a friend recommended you, will produce a distribution that flatters your advertising. People pick the nearest thing or pick other, and either way the resulting chart is wrong in a direction you will not detect.
The second is order. The first two options in any list are chosen more often than their merit warrants, particularly on a phone where the rest require scrolling. If the order is arbitrary, that bias is noise. If you have put your preferred answer first, it is not noise, it is a thumb on the scale.
Other is a signal, not a category
Most forms treat the other option as a tidy way to cover the gaps, and then nobody reads what people typed into it.
A high rate of other is one of the clearest diagnostics a form produces. It means your list is missing something that matters to a real share of respondents, and the text they typed tells you exactly what. Options that appear repeatedly in the other box are options that should be in the list, and promoting them improves both the data and the experience.
This is worth checking quarterly on any question you report on. Option lists are written once against the business as it was, and businesses change faster than their forms do.
Ask about behaviour, not intention
People are reliable witnesses to what they have done and unreliable forecasters of what they will do. Any question phrased as would you or how likely are you to produces answers that are systematically more positive than the eventual behaviour.
Where you can, ask about the past instead. How many times have you used a service like this in the last year is answerable and roughly accurate. Would you use this service in future is a question about self-image, and most people answer it generously.
This matters most for anything you are planning around. A list of people who said they would be interested is not a forecast, and treating it as one has caused more misdirected effort in small businesses than almost any other data mistake.
The one thing to do
Read your form out loud, as though you were asking a customer across a counter. This single exercise catches most of what is in this article, because the problems are all things that sound wrong when spoken and look normal when written.
Leading questions sound like fishing. Double-barrelled questions make you pause halfway. Internal jargon sounds absurd out loud, because nobody has ever said the words enquiry type to another human being. And unbalanced scales become obvious the moment you read the options in sequence and notice there is nowhere to put a bad experience.
It takes three minutes per form and it needs no expertise at all, which makes it the highest-value thing on this list by a considerable distance.
Do it with somebody else in the room if you can, because the questions that make you hesitate while reading are the ones you will otherwise talk yourself past. A listener who says what do you mean by that has just found the question to rewrite.