STC Usability SIG Newsletter
Usability Interface

Methods and Guidelines to Avoid Common Questionnaire Bloopers

By Chauncey Wilson

Over the years, I’ve often heard colleagues say "let’s throw a questionnaire together and find out what our users think about our product". Implicit in this statement is the assumption that questionnaires are easy to design, administer, and analyze. This assumption is far from the truth.

The design of questionnaires involves social processes (collaboration among stakeholders), persuasive processes (getting respondents to answer the questions), business processes (do I have the questions and response categories that will yield data to help us answer our business questions), cognitive processes (understanding about how memory and context affect respondents answers), and analytic processes (how do I analyze and present the data). Throwing a questionnaire together is at best a waste of time and at worst, a source of flawed data that could affect your company’s reputation and revenue. In this short article, I will present a set of methods and principles that will help you avoid the most common questionnaire bloopers.

Apply the basic rules of user-centered design to the design of questionnaires

Design a questionnaire in much the same way that you would design a product (except that your design cycle might be in days rather than weeks or months). Start by gathering requirements from your stakeholders. What do your stakeholders want/need to know about users and the product? Follow the requirements gathering with a clear definition of goals and an explicit statement of how to build trust and provide respondents with benefits that outweigh the costs of filling out the questionnaire. Conduct prototype reviews and iterative testing. Ensure that you have a data analysis plan so that you understand what to do when all the data pour in.

Gather Requirements and Questions from Stakeholders

• Interview key stakeholders about what they know and don’t know about users and how they use a product. Stakeholders include actual users, sales, marketing, development, documentation, training, senior management, and technical support.
• Brainstorm with the product team about what they want to learn from a survey. If you can do this as part of a regular product team meeting, you can get many of the stakeholders in the room at the same time.
• Distribute 3x5 cards at meetings or individually and ask stakeholders from different groups to write 1-3 questions that they would like to ask users. Avoid asking for too much here. This technique can be useful for getting insight into what issues are most important for different groups.
• Conduct a short brainwriting session. Brainwriting is an extension of brainstorming where each person writes a question on a card and then passes it on to the next person who then reads the previous question and passes the card on to the next person who sees the first two questions and adds a third question. The premise is that seeing the questions of others will prompt additional relevant questions. This can be done in about fifteen minutes at team meetings and yield a large selection of questions.
• Conduct a focus group to find out what issues are important to key user groups. Although focus groups are often derided by usability experts as weak sources of design information, they can be an excellent source of requirements for questionnaire design. The more open-ended nature of a focus group can provide input for more structured online or paper questionnaires.

Be Explicit of the Goals of Your Questionnaire

Too often, the goals of a questionnaire and each question on the questionnaire are not clear. Is the purpose of the questionnaire to understand customer loyalty issues, gather information before or after a usability test, gather requirements, understand why the product is not selling as expected, or understand how people use the product? I would recommend that each question on a questionnaire related to a specific business goal and user experience issue.

Consider How to Establish Trust, Increase Rewards, and Reduce Social Costs for Respondents

You can design your questionnaire to create trust among respondents and influence the respondent’s expectations about the benefits and costs associated with filling out the questionnaire. Don Dillman’s classic book, Mail and Internet Surveys The Tailored Design Method, Second Edition (2000, p. 27) notes that you can increase trust in the questionnaire by:

• Providing tokens of appreciation in advance (though be careful not to make the tokens too large since this may be a source of bias)
• Indicating clearly that the sponsor is legitimate and can do something with the results
• Making the questionnaire appear important
• Indicating how the data will be used

Dillman’s suggestions for increasing rewards to respondents include:

• Design an interesting questionnaire
• Use positive language that makes the respondent feel like a collaborator
• Provide tangible rewards
• Thank the user for helping
• Ask people for advice

Suggestions for reducing the costs of completing a questionnaire include

• Make the questionnaire usable,
• Avoid embarrassing questions (don’t ask "how old are you?"),
• Minimize the need for personal information,
• Make every question is relevant and avoid lengthy questionnaires,
• Allow users to change answers easily in online surveys.

Create Prototypes of the Questionnaire and Review Against Principles of Survey Design

Design a prototype questionnaire, including the cover page, and compare it with the principles of questionnaire design. These principles should cover language, relevance, page layout, response categories, and ordering of the questions. I recommend that the questionnaire designer ask four people to review the questionnaire, and that you interview a few people not closely associated with the project as they read the questionnaire and think aloud about their reactions to it.

Devise a Data Analysis Plan

A common error in designing and implementing a questionnaire is to not devise a data analysis plan that spells out how answers will be coded (for example, how will you code non-responses, unusual responses, or ratings where people circle two numbers when you only want a single answer), what analyses you will do on single questions and sets of questions, and any hypotheses that you may have and what questions will be used to test those hypotheses. You should do this even if you have survey software that does an automatic analysis of the data. You might find that your automated software doesn’t allow some of the analyses that you need to answer the questions that are important to your stakeholders.

Conduct Limited Testing of the Questionnaire With Actual Users

Get a small sample of users (or people as close to the expected users as possible) and have them fill out the questionnaire under realistic conditions and give you feedback. Make your final changes based on this input and do a final edit.

Principles of Questionnaire Design

• Ensure that your first question is relevant to everyone, easy, and interesting.
• Avoid vague response quantifiers when precise quantifiers can be used.

In example 1a, the response categories are vague and can be interpreted differently by respondents. The data from this question would be nearly impossible to interpret. Example 1b eliminates the vague quantifiers with more specific answers.

1a. How often did you Product X during the last month? (Check 1 answer)
__ Never
__ Rarely
__ Occasionally
__ Regularly

1b. How often did you use Product X during the last month? (Check 1 answer)
__ Not at all
__ 1-3 times a month
__ Once a week
__ Two to four times a week
__ Once a day
__ More than once a day