STC Usability SIG Newsletter
Usability Interface

Market Research and Usability

by Caroline Jarrett

I am a usability consultant and I believe, and find in practice, that usability evaluation is the best way to find out whether a document works for its users. However, I have frequently been in a position where my clients have decided to "do market research" because they believe that they have a problem document and they think "we’ll ask our customers about it."

Usability evaluation is one type of market research but it is not the most frequently used technique. In this article, I introduce some of the more common techniques and what you might expect to learn about a document by using them. I’ll use the example of a car owner’s manual–the small book that comes with a car, and explains things like how to change the oil and what the switches mean.

Traditional Market Research Techniques

Traditional market research usually divides into quantitative and qualitative techniques.

Quantitative techniques

Quantitative techniques aim to achieve statistically reliable views by interviewing a sample from your target population. The sample size is often 250 or more.

They are based around the questionnaire. To be effective, you must determine which questions you want to have answered, what the likely responses might be, how the responses might be grouped into the target audience, and an order of asking the questions that will seem sensible to your respondents. Your questionnaire goes to interviewers who walk the streets asking reluctant people to answer.

Or, even worse, they conduct telemarketing and disrupt people’s lives for you. In either case, if you are respectful of people’s time and trouble you will get good results for very simple questions. You can get statistically reliable views of the general public or of large subsets of the general public (such as men over 50 who own convertible cars), and probably some anecdotal background information.

Typical responses would be:

• The number of car owners who will change their car within the next 2 years
• The number of car owners who ‘never or rarely’ refer to the owner’s manual

Quantitative techniques are best for generally reliable public opinion, and the wider picture–overall impressions rather than detail. Quantitative research is also useful if you have a very short task time, for example, to decide to scan or throw away the manual. You can ask the customer to try the task as part of the interview.

Limitations:

• Difficult to construct an informative set of questions short enough to get a good response rate.

• Lack of depth. Why don’t the car owners refer to their manual? What will prompt the owner to change the car?

Qualitative Ttechniques

Qualitative techniques use much smaller samples than quantitative techniques. The qualitative researcher spends more time with each person, and explores why people make certain choices. The classic technique is the focus group. A moderator gets the group to discuss their lifestyles, and their feelings, thoughts, and impressions of the product.

The product may be known to the group (the manual for a specific brand of car) or they may be asked to discuss the general group of products, such as how they might use a car owner’s manual, and its significance in their choice of car. The moderator may use "stimulus material" such as an example manual, or perhaps a conceptual display of images related to cars and manuals. Sometimes these groups meet over a long period, forming a "cohort" or reference group to try various versions of the product.

A related technique is the "depth interview," which is often used by quantitative researchers to formulate their attitude statements for their questionnaires. Depth interviews are similar to focus groups, but are conducted one-to-one, or perhaps in pairs (maybe husband and wife) together with the interviewer.

Typical results:

• Identifies the issues that should be explored in a large-scale technique.
• Allows interviewers to explore the reasons for the impressions gained, such as "I don’t read the manuals because I am too busy" or "I look at it once and then put it aside until I need it."
• Explores overall impressions of the product, the aspects the user recalls and reports.
• Compares ideas about the product type in general, for example, reasons why the group prefers Renault advertising to Ford advertising.

Limitations

• Relies on the user recalling and understanding the way they interact with the product.
• Requires a skilful facilitator / interviewer to ensure that the most vocal people do not ‘drown out’ the views of quieter participants.
• Focuses on comments on areas which the participant would not encounter in normal use of the product (participant: "the section on air-conditioning is confusing" facilitator: "does your car have air-conditioning?").

What happens next

My experience of quantitative and qualitative techniques has been that they often tell you that you have a problem with your document, product or service, but they are less useful at pinpointing what to do about it. For example, I worked with a client to research a particularly large and complex form–over 1000 boxes. The quantitative research told us the customers did not like it. The focus groups told us that they thought it was too large, too complex, too confusing. But we had to ask those 1000 questions–so what could we change?

I persuaded the client’s project manager and designer to observe a usability evaluation. Now they could see which instructions were not working and understand which phrases were confusing. Worse, the users just didn’t get a key idea about the structure of the form. But because they failed to understand it, they wouldn’t have been able to tell a telephone interviewer what had confused them.

So if someone says to you "I want to do market research to find out what my customers think of this product," I’d reply: "By all means. But do some usability evaluation as well so that you find out what it is about using the product that makes them think that."

Further information

To get more information on traditional market research techniques: ESOMAR, the World Association of Opinion and Marketing Research Professionals, lists organizations in 100 countries on their web site at www.esomar.nl