Usability and User Experience Resources
Idea Market Afterthoughts 2004

How many users does it take for a (web) usability test?

Activator: Carol Barnum

5 starter questions:

1. What is your testing goal? (You need to know this first to answer the other questions)
2. How many users do you need to reach your goal?
3. How do you define your users?
4. What’s the basis for small numbers? How small is OK?
5. The numbers controversy: Do small numbers work for web usability testing?

Testing goals:

• To find problems with the product
• To eliminate problems with the product
• To improve user satisfaction
• To determine if you made the right assumption about user tasks
• To help developers understand the user experience
• To learn from prototypes
• To determine the level of proficiency (e.g. time on task)
• To understand the Learnability of the product
• To determine quantitative issues

Numbers needed to reach goals:

• For summative (product completion testing, where statistical validity is needed) Need 25 to 30 or more users
• For exploratory testing, can use Nielsen model of 4 to 5 users (even as small as 3 users)
• Numbers depend on the kinds of users (users must match in experience, knowledge)
• Must select specific subset of user population; must use scenario-based tasks; for diagnostic purposes; part of iterative process

Controversy--Why web testing may be different:

• Jared Spool tested ecommerce sites
  • Users given money with goal to find something to buy (open ended task)

• Rolf Molich directed comparative usability evaluation of Hotmail
  • 9 different labs set up different goals and users
• Both studies showed that it takes many more users than 5

• Use scenarios
• Screen users to match specific subset of user population

• Random sample with small numbers
• Common skill sets of users (e.g. level of domain knowledge)
• Same motivation from users (desire to reach task goals)

• Data may not be generalizable—not applicable to any user
• However, findings for different tasks can point up the same problem
• Issue may be information architecture

What types of users to recruit?

• It depends on business goals.

• If numbers are very small, try not to mix the users.
• Better to go for one end or the other on the continuum; avoid the middle.
• "Most people" is not a good description of the user.
• Should you separate the documentation from the product?
• Test documentation by itself
• Test product without document to see where users need it
• What to do with one user’s findings (outlier data)
• Examine the issue for severity
• Study the screener for particulars about this user
• Ask yourself: can you envision other users having this problem?
• Have you seen this in other studies?
• What is the importance of this user (validity—a big customer)?

How to validate the findings with small numbers

• Talk to tech support
• Talk to training
• Talk to sales/marketing support (but be cautious, because they may not be talking to users)
• Know what the product is supposed to do
• Know what the product can deliver

Reporting results

• Capture the "eureka" moment
• Shun percentages when sample size is small (say, instead, "several" users or "three out of five" users)
• Create a matrix of tasks
  • Indicate success/failure
  • Time
  • Questions asked
  • Sample comments
  • Errors and severity (recovery from errors should also be noted)
• Distinguish confidence levels in findings
• Descriptive information is also useful (e.g. user confusion)