Methods and Guidelines to Avoid Common Questionnaire Bloopers
By Chauncey Wilson
Over the years, Iíve often heard colleagues say "letís throw a questionnaire together and find out what our users think about our product". Implicit in this statement is the assumption that questionnaires are easy to design, administer, and analyze. This assumption is far from the truth.
The design of questionnaires involves social processes (collaboration among stakeholders), persuasive processes (getting respondents to answer the questions), business processes (do I have the questions and response categories that will yield data to help us answer our business questions), cognitive processes (understanding about how memory and context affect respondents answers), and analytic processes (how do I analyze and present the data). Throwing a questionnaire together is at best a waste of time and at worst, a source of flawed data that could affect your companyís reputation and revenue. In this short article, I will present a set of methods and principles that will help you avoid the most common questionnaire bloopers.
Apply the basic rules of user-centered design to the design of questionnaires
Design a questionnaire in much the same way that you would design a product (except that your design cycle might be in days rather than weeks or months). Start by gathering requirements from your stakeholders. What do your stakeholders want/need to know about users and the product? Follow the requirements gathering with a clear definition of goals and an explicit statement of how to build trust and provide respondents with benefits that outweigh the costs of filling out the questionnaire. Conduct prototype reviews and iterative testing. Ensure that you have a data analysis plan so that you understand what to do when all the data pour in.
Gather Requirements and Questions from Stakeholders
Be Explicit of the Goals of Your Questionnaire
Consider How to Establish Trust, Increase Rewards, and Reduce Social Costs for Respondents
You can design your questionnaire to create trust among respondents and influence the respondentís expectations about the benefits and costs associated with filling out the questionnaire. Don Dillmanís classic book, Mail and Internet Surveys The Tailored Design Method, Second Edition (2000, p. 27) notes that you can increase trust in the questionnaire by:
Dillmanís suggestions for increasing rewards to respondents include:
Suggestions for reducing the costs of completing a questionnaire include
Create Prototypes of the Questionnaire and Review Against Principles of Survey Design
Design a prototype questionnaire, including the cover page, and compare it with the principles of questionnaire design. These principles should cover language, relevance, page layout, response categories, and ordering of the questions. I recommend that the questionnaire designer ask four people to review the questionnaire, and that you interview a few people not closely associated with the project as they read the questionnaire and think aloud about their reactions to it.
Devise a Data Analysis Plan
A common error in designing and implementing a questionnaire is to not devise a data analysis plan that spells out how answers will be coded (for example, how will you code non-responses, unusual responses, or ratings where people circle two numbers when you only want a single answer), what analyses you will do on single questions and sets of questions, and any hypotheses that you may have and what questions will be used to test those hypotheses. You should do this even if you have survey software that does an automatic analysis of the data. You might find that your automated software doesnít allow some of the analyses that you need to answer the questions that are important to your stakeholders.
Conduct Limited Testing of the Questionnaire With Actual Users
Get a small sample of users (or people as close to the expected users as possible) and have them fill out the questionnaire under realistic conditions and give you feedback. Make your final changes based on this input and do a final edit.
Principles of Questionnaire Design
In example 1a, the response categories are vague and can be interpreted differently by respondents. The data from this question would be nearly impossible to interpret. Example 1b eliminates the vague quantifiers with more specific answers.