You know how many unique visitors come to your company’s web site. You know what they click on. And you know what and how much they buy, or download, or watch. But you also want to know what they think or feel about your site, and so you’re considering conducting an online survey. How do you go about asking the right question?
When we talk about the “right” survey question we’re entering the territory of validity: does the measure you use reflect the real-world meaning of the concept under consideration? (from Babbie, E., The Practice of Social Research).
Notice that this definition has two main requirements: (1) that you know the “concept under consideration,” and (2) that the way you measure that concept reflects its “real-world meaning.” Meeting those requirements involves both art and science. It’s both a subjective and objective process, with several choice points along the way. Here’s a guide that can help you determine the “right” question to ask.
Step 1: Take note of your company’s most pressing questions
Surveys usually emerge from a point of tension or a felt need. Something isn’t well understood; something could help guide business decisions, if only it were known; assumptions have been made about customers or their experiences, but you’re not sure if they’re accurate. Whatever it is that gives impetus to the idea of a survey, start to write down, in your common, everyday language, what it is you want to know.
For example, suppose you want to know about customers’ experiences of looking for products on your web site. You’ve come up with the following questions you’re considering:
1. Did you find the product(s) you were looking for during your visit to our site today?
2. How satisfied were you with the products we offer on our site?
3. How happy were you with the products we offer on our site?
Step 2: Begin to force some distinctions
Sometimes we can focus in on what we want to know by gaining clarity about what we don’t want to know. Do you want to know about something more on the attitude level, like “satisfaction” or “happiness,” or are you more interested in people’s accounts of their experiences (“I found it,” or “I did not find it”)? Force distinctions by writing alternative versions of your question. Try the different version out on yourself and others. Which one seems to come closest to what you’re trying to get at?
A quick review of the questions above tells us that they’re all exploring customers’ experiences of products on your web site, but they’re asking about those experiences in different ways. In this case, the “right” question is mostly a matter of which, if any, of these three questions most closely matches your intentions. Which is the best fit for the concept in which you’re interested?
Question 1 gets at the success of customers’ product searches (and assumes they were looking for a product). Reviewing this question may make you realize that you’re actually more interested in “how easy” it was for customers to find what they were looking for, not just whether or not they found it. So you might add a fourth question to your list:
4. How easy was it to find the product you were looking for during your visit to our site today?
Questions 2 and 3 are more about how customers felt about the products on your site. They make an interesting distinction: Question 2 uses a term more commonly found in surveys, “satisfied,” while Question 3 asks about “happiness,” less-often used in surveys, but probably a higher standard to reach (maybe your business has as one of its core values that customers should be “more than satisfied” … you want them to be “happy”). Further, in reviewing these two questions you realize that both of them are focusing on the products on your site, but not on the customers’ experience of finding those products. So you write down two more questions to consider:
5. How satisfied were you with your search for products on our site today?
6. How happy were you with your search for products on our site today?
Step 3: Consider your wording options in the light of your business needs or objectives
If you want your research to be put to good use, there better be a close association between the needs of the company and the research you’re conducting. Ideally, for each question on a survey, there should be some understanding of how the data gathered from that question can make a difference to the business.
This is a good time to take a step back and ask a couple of foundational questions.
- Why does it matter, or, what difference will it make to our business if we know the answer to this survey question?
- What will we do with the information we gather?
These foundation questions will help to keep you grounded as there is no end of what you could learn from surveys. But asking questions like the two above brings you back “down to earth.” They require you to “ground” your curiosity in questions with practical relevance.
So practically speaking, do the business needs of the company lead us to be more interested in the customers’ views of our product selection, on the ease of finding those products, or on their satisfaction or happiness with the products or the overall experience? Or, do all of these seem to be missing the mark? (If so, then head back to Step 1.)
Step 4: Use everyday, non-technical language
After using the lens of business needs, it’s good to get back to the customer. Try to put yourself in your customers’ shoes. What wording is most likely to resonate with them? For example, what word would they be likely to use to describe a good experience on your site? Would it be “good,” or “successful,” or “satisfactory”? Or would they describe it as “no-hassle,” “intuitive,” or “easy”? “Fun,” “fast,” or “friendly”? Do they think of the things you sell as “products,” or are they more likely to call them “shoes,” or “equipment,” or “services,” or “books”? Even if you’re writing for a “technical” audience, they will have their own everyday language. The more your questions reflect that language, the likelier you are to get valid responses.
Step 5: Try it out
“Try it out” doesn’t mean formal A/B Testing of your questions (though if you have the time and budget for it, have fun), but trying it out on a variety of people. Show them the question and ask them what they think you’re asking. If they’re confused, or if there’s a lack of consensus, then it’s probably time to clarify and simplify.
