STC Usability SIG Newsletter
Usability Interface

How the Usability SIG Survey Was Developed

by Naomi A. Kleid

When the leaders of the STC Usability SIG wanted to run a survey of its members, I volunteered to design and deploy the survey. At about the same time, I also volunteered to work with my local STC chapter to run its Employment and Salary Survey (see articles describing the creation of this survey and presenting some of the findings in the September 2003 issue of Carolina Communiqué at www.stc-carolina.org). Although I had extensive experience creating surveys and analyzing survey results, working on these two surveys taught me a lot about a new survey tool.

While the Carolina chapter wanted to do an online survey, the Usability SIG survey did not start with that intention. We considered doing a mail survey by including the questions in the January issue of Usability Interface. However, the costs of mailing to the membership and having the membership mail the survey back, and the tasks associated with designing the survey pages and tabulating all the responses by hand argued against a mail survey.

Getting Started

As I investigated running an online survey for the Carolina chapter, I learned that for $19.95 per month, I could have a professional subscription at SurveyMonkey.com that would permit me to set up, run, and take down the data from both surveys concurrently. The results would be tabulated separately, and delivered both in HTML files for display purposes and in spreadsheet files for further data analysis. Both organizations agreed to run their surveys during the first quarter of 2003, so I split the three-month cost between them.

The Usability SIG survey development followed the procedure outlined by Chauncey Wilson in the January 2003 issue of Usability Interface (see: "Methods and Guidelines to Avoid Common Questionnaire Bloopers").

To start, I asked the SIG leadership the following questions, to help identify the goals of running the survey:

• What are the purposes of the survey?
• What are your key questions?
• What things / information would you like to know as a result of the survey?
• What decisions are you grappling with?
• What kinds of changes might you consider making after you get some data?

The purpose of our survey was to identify the audience of the newsletter, what they wanted for content and distribution method, and how the newsletter could increase SIG membership and satisfaction. While these answers appeared to focus on the newsletter, they also suggested an interest in getting feedback on the Usability SIG, itself, and other SIG offerings such as the web site, e-mail discussion group, usability sessions and SIG sessions at the annual conference, and possible training options. These issues suggested a broader scope, one that was consistent with the role of online surveys as defined by Dr. William MacElroy, also in the January 2003 issue of Usability Interface (see: "The Role of Online Surveys in the Usability Assessment Process").

At the same time, I was impressed with Charlie Kreitzberg’s concerns about so many usability professionals struggling to stay working in the usability field (see: "Crisis in the Profession" in Usability Interface, October 2002). I was influenced by my work with the Carolina Chapter survey and our effort to evaluate the impact of our local economic downturn on the employment situation for our STC members. Consequently, I wanted to include questions on demographics and employment status for Usability SIG members.

The survey grew to cover five topics (below), and offered respondents the opportunity to skip over questions or sets of questions if they wanted:

• SIG (satisfaction and member requirements),
• Newsletter (satisfaction and requirements),
• Web site (satisfaction and requirements),
• Comparisons / rankings for the SIG’s offerings,
• Member information (demographics and employment status).

An early draft of the survey consisted of 53 questions mostly requiring answers in multiple-choice or numerical formats. In the end, the survey was edited to 20 questions with no branches.

A number of trade-offs made this questionnaire simplification possible. In each case, the decision opted for letting SIG members express their wishes as fully and completely as possible. For example, one trade-off was to accept an increase in analysis time (required to work through the write-in responses) in order to save respondent time (required to work through all the detailed questions and response options). By decreasing the time required from each respondent, we hoped to increase the response rate and, consequently, get thoughtful feedback from more SIG members. Another trade-off was to accept a wide variety of write-in responses to each question (again requiring more analysis time) in order to get a full range of possible answers, rather than to quantify responses to a limited set of largely predefined answers.

Using SurveyMonkey

I took out a one-month professional subscription to SurveyMonkey using the online form at their Web site. Within minutes my account was automatically activated, and I started entering both surveys. Although I had never entered an online survey before, I entered all the questions for both the Usability SIG survey and the Carolina chapter survey in one day—that amounted to 20 questions for the Usability survey and 64 questions with several branches for the Carolina survey in just a few hours.

Had either survey been 10 questions long or less and had we expected no more than 100 responses per survey, I could have taken a basic subscription with SurveyMonkey and run the surveys at no cost. The basic subscription does not offer some advanced features, such as requiring an answer to a question before respondents can proceed to the next question, branching over groups of questions that are inappropriate to a specific respondent, filtering sets of responses based on specific questions (to aid in the analysis of results), and managing the appearance of the survey by including an identifying logo or customizing the colors of the survey. For readers interested in trying an online survey, I recommend going to www.SurveyMonkey.com and creating a free survey.

When the Usability SIG survey was online, we conducted a usability test of members of the Usability SIG team to evaluate the question and answer formats, usability of the survey, and length of time it took them to complete the survey. We fine-tuned the welcome and some of questions. When the survey was ready for distribution, I clicked on the SurveyMonkey Collect function icon. SurveyMonkey created an HTML link giving a unique hyperlink location to announce the survey and direct respondents to it. My job was almost done; all I had to do was take the data files down at the end of the survey and send the files to the Usability SIG leadership team for analysis.

The responses are presented in Results of the STC Usability SIG Membership Survey.

Disclaimer: Neither STC or the Usability SIG endorse any survey service.