Usability and User Experience Resources
Idea Market Afterthoughts 2004

How can we ensure rigor in participant recruiting, and why is it important?

Activator: Deborah Hinderer Sova

Recruiting is the foundation upon which a good usability evaluation is built and subsequently succeeds. To put it succinctly: sloppy recruiting = shaky foundation = faulty data (or at least less reliable data).

Feeder Questions

• How do we tout participant recruiting as the foundation for user testing?
• When will "just anyone" do? Or won’t they?
• When do we need someone specific?
• What examples do we have of successes and failures in using "just anyone"?
• What can we do to strengthen the [recruiting] foundation of the evaluation triangle?
• What are some specific ways in which we can seek out the right users?
• How do we ensure users fit the target profile?
• What role can documentation play in participant recruiting?
  

Actual Idea Market Discussion Focus

Attendees mainly were technical writers, not usability specialists or participant recruiters. They had not conducted usability evaluations—for products, websites, or documentation. Therefore, attendees were not familiar with recruiting participants for evaluations, let alone the challenges of finding the appropriate users for testing. In fact, nearly all attendees said that they most often write technical documentation without ever directly working with or talking to users.

Shift in Direction

To better center the discussion around the actual experience of the attendees, the activator shifted the focus of the discussion instead to how technical writers:

• Might get user feedback on documentation
• Currently gather requirements for documentation and incorporate users’ needs into the documents they write for users
• Would test usability, if there were time and budget
  

Getting Feedback on Documentation

Attendees said that their companies (or their managers) see usability testing as adding time, not value, to a documentation project. Therefore, time for testing documentation is not built into their writing schedule.

The activator then asked attendees: If you could test documentation, with whom would you test—just anyone, or someone in particular—to get the kind of feedback on which you would confidently make changes to the document?

Anyone

Attendees said anyone—perhaps even the person in the next cube—could provide decent feedback on:

• Presentation—are text and graphics visually inviting?
• Consistency—are the index and table of contents consistent with the content and design of the document?
• Simple procedures and forms—are the steps clear?
• New knowledge—everyone is on a level playing field
  

Someone in Particular

Attendees said that feedback will be "skewed" if not provided by people who "are in the job" or who at least have something in common with the system that is the subject of the documentation.

According to attendees, areas where feedback is needed from specific people include:

• Content—quality and especially context
• Specific user tasks—what are they trying to accomplish?
• Complicated procedures and forms—where special knowledge is required to continue with steps
• Training required to use overall system
• System assumes repeated, continual use
• System assumes only occasional use
  

Undesirable

Attendees acknowledged that feedback from certain people would be undesirable:

• Beginners, or novices—unless the system being documented is easy to use and requires no training
• Authors—because being too close to the document, they may:

• Not recognize gaps
• Favor their own biases
  

Documentation Requirements

Again, most attendees said they did not get to talk to or work with users directly. They described that they get document requirements from:

• Management
• Tech support logs
• Putting themselves in the users’ place
  

Management

Attendees said that management most often provided:

• The basis, or rationale, for business decisions
• Information on security issues
• User metrics (and it’s not always clear just where management got the information on which the metrics are based)
  

Tech Support Logs

Email complaints to the support desk and technical support logs can be useful for learning about user problems after the fact, such as difficulties caused by changes made to the system. However, attendees said that technical communication managers often filter complaints. Managers must go into "defense" mode, prioritize complaints, and ask the writers to "fix" the system problems that feasibly can be alleviated in the documentation.

Putting on the User’s Hat

Attendees said that they often resort to imaging themselves as users when designing instructions. However, they acknowledged the following problems with that process:

• Different users may use different parts of the same interface

• What are people using that we don’t know about?
• Which people are doing the same things?

• It’s too easy to make assumptions about users that might be wrong
• It’s difficult to understand the needs of novice users—they’re not me
  

Therefore, writers try:

• Breaking down material by beginner, intermediate, and advanced, based on what they think they know of the users
• Tracking informal processes they hear of
• Attending annual conferences, such as STC
  

When asked to think of ways that they might get information more directly from users, attendees suggested:

• Setting up a booth at a local business conferences that their users would likely attend

• Ask users to fill in needs surveys
• Give them "toys" as a reward for participating

• Send surveys to customers who purchased the previous version of a system
• Send surveys to users once something is changed
  

If Technical Writers Could Test

Attendees acknowledged that as technical writers, they are responsible for seeing the whole picture and seeing it early. They discussed what they would do if they had time and budget for testing documentation and/or the system being documented.

• Do usability tests early on
• Find out what’s broken with the system so documentation can compensate—engineers design what’s "sexy", not necessarily usable
• Learn what the system does and does not do
• Learn what users do and what they do not use
  

One attendee who was allowed to observe a usability test cited the benefits of her observations:

• There were 7, not 3, distinct user groups
• The system needed more categories, or views
• Users wanted task-based documentation
  

Conclusions

• Writers putting themselves in the users’ place is always a good thing, but not enough
• Obtaining information from tech support logs and users surveys helps improve documentation
• Usability testing with actual users early on is ideal for creating good documentation