STC Usability SIG Newsletter: Usability Interface
Usability Testing of Documentation has Many Benefits of Unknown Value

by Elaine Ostrander
A full report, expanding on a short article in  Usability Interface, Vol 7, No. 2, October 2000

Study shows technical communicators find many benefits in usability testing of documentation, but can’t quantify them.

A pilot study was conducted by a University of Houston graduate student on the costs and benefits of usability testing of documentation. The purpose of the study was to compare the costs versus the benefits of usability testing of documentation so that an average return on investment could be calculated. Participants, however, while they attributed many benefits to usability testing of documentation, could not, for the most part, produce even guesstimates for how much time or money was saved. The study failed in its intent to document a persuasive ROI that could be used to convince otherwise unwilling management to initiate usability studies. The study instead turned up the disappointing fact that technical communicators simply aren’t keeping track of the essential figures. There were, however, promising clues that the impact of usability testing of documentation may be enormous if it could be quantified.

What was studied

In this study, only usability testing of documentation was considered, not software or product usability. Formal laboratory testing was NOT considered, but only informal, over-the-shoulder style usability testing.

Participants

The eleven participants were volunteers solicited from the Usability Special Interest Group of the Society for Technical Communication. Participants held various jobs relating to usability testing of documentation. Participants reported having participated in from 3 to 100 or more usability tests, the average number being 38 usability tests. Participants had an average of over 10 years’ experience participating in usability testing, with the range being anywhere from one to 22 years. Participants reported that usability testing consumed up to 30 percent of their present jobs, with the average being 12 percent of their present jobs. Participants were identified only by their e-mail addresses, and not by age, race, or gender. At least four different countries on three different continents were represented, with the majority of the participants replying from the United States.

Method

Participants were given a list of eight stages in the process of usability testing and asked to fill out a table indicating the minimum, maximum, and average amount of time spent on each stage. They also indicated the minimum, maximum, and average hourly cost for each stage.

Participants were given a list of proposed benefits of usability testing, and asked to indicate if an item was found to be a benefit in their experience with usability testing. They were also asked to estimate the "percentage of time or money saved/added to/from" the project as a result of each benefit.

Findings about cost

Determining the cost of informal usability testing of documentation was fairly forthright. Results are shown below:

Table 1. Cost of Conducting a Usability Test of Documentation

Table 1 (above) shows the average values reported by all participants, for the minimum, maximum, and average time and hourly costs for each stage of usability testing. The times and hourly costs were multiplied to get the calculated values for minimum, maximum, and average cost. We learn that the different stages require varying degrees of labor and that the total cost can vary widely, but usually runs between eight and nine thousand dollars.

Findings about benefits

Findings about benefits were not as forthright. Participants freely checked off items they believed were benefits, but most could not quantify the amount of benefit. Therefore, in these findings the benefits and their values are reported separately.

Benefits

Votes for each benefit are tallied and ranked in the table below:

Table 2. Ranked Benefits of Usability Testing of Documentation

The data in the above table demonstrate that participants unanimously agreed on these benefits of usability testing of documentation: 1) "Decreased need for customer support to augment documentation"; and 2) "Costly errors created by poorly written instructions are eliminated." Nearly all participants (91 percent) agreed on these benefits of usability testing of documentation: 1) "End-user completes tasks faster using improved instructions"; and 2) "End-user accesses information faster in final document." Most participants (73 percent) agreed on these benefits of usability testing of documentation: 1) "Increased customer satisfaction and loyalty due to quality of final document"; 2) "Decreased need to rework before final publication"; and 3) "Decreased need to publish after-the-fact fixes."

Agreement among participants began to dwindle after this. The remaining benefits are probably not realized in some situations, that is, either this is not a always a benefit or the participants were not aware that the benefit resulted: 1) "Document completed in less time due to better focus on user needs"; 2) "Decreased need to publish further editions of document"; 3) "Increased sales of product that document accompanies"; and 4) "Increased sales of finished document."

The reasons for this lack of reported benefits may not be that the benefits are less, but that participants were not able to notice them. For example: 1) The document may be completed faster than it would have been, but the time saved is impossible to estimate; 2) Further editions are still necessary because the products the document accompanies are revised; 3) Participants were not informed about the sales figures of the products that the document accompanies; and 4) Technical documentation is usually not sold by itself, but as a part of a product/service combination; therefore, it is impossible to isolate such a benefit as "increased sales of finished document."

 Value of Benefits

The majority of the participants could not fill in the column requesting the "percentage of time or money saved/added to/from project." When it came to quantifying the amount of benefit, only 21 percent of the quantification questions were answered at all. Participants who declined to quantify made comments such as "too subjective," "can’t quantify," "too hard to even estimate," and "haven’t really gathered this data," or simply left these questions blank.

Therefore, in Table 4, below, in addition to ranking the benefits and putting the percentage of time or money saved, the number of responses per benefit are also listed.

Table 4. Percentage of Time or Money Saved for Benefits of Usability Testing of Documentation

Table 4 (above) shows the amount of time and money participants attributed to each benefit. Since response was low for this item, the table discloses the number of participants who responded for each benefit. This allows the reader to evaluate how much validity to give to the final number. For example, an added value of one thousand percent was attributed to the benefit, "Increased sales of product that document accompanies." However, only one participant responded. The participant’s testimonial is superlative, but it is only one opinion. On the other hand, for the benefit, "Decreased need for customer support to augment documentation," five participants responded, attributing an average added value of 32 percent. This is a figure which we can have more trust in since several people contributed to it.

Table 4 (above) demonstrates how clearly one participant’s attribution of the added value of a single benefit ("Increased sales of product that document accompanies.") was enormously higher than all other attributions. One benefit ("Increased sales of finished document") had so little relative added value that it is almost invisible on the graph. All other benefits seem to be in the same general range, that is, participants attributed approximately the same general range of added value (14 to 41 percent) to the benefits.

Impact of Benefits

From the previously reported benefit figures, an attempt was made to estimate the impact of each benefit. To calculate impact, first the percent of participants who agreed upon each benefit was determined. Second, the reported quantity of benefit was averaged. (This value was admittedly flawed because so few participants were able to report any quantity at all.) Then the two values were multiplied and the result was called the impact of the benefit. The impact figure shows each benefit’s relative impact, considering this amount of benefit could be expected to occur at the given percentage of consistency. Results are shown in the table below:

Table 7. Impact of Benefits of Usability Testing of Documentation

For item 1, "Increased sales of product that document accompanies," the impact is enormous. This would mean a very positive ROI, and shows very promising results for a future study in which benefits are carefully tracked and tabulated. The high degree of impact is demonstrated for all items except the last one, item 11. For item 11, "Increased sales of finished document," only 27 percent of participants agreed this is indeed a benefit of usability testing of documentation. Furthermore, since not a single participant responded with a quantitative figure for the added value of the benefit, this benefit scored zero, or no impact.

The graph below shows the impact findings visually:

Figure 8. Impact of Benefits of Usability Testing of Documentation

Figure 8 (above) shows visually the impact per benefit. The visual result looks very similar to Figure 5, "Amount of Time and Money Save By Each Benefit of Usability Testing of Documentation," with the same item number one item being overwhelmingly above the others, and the same last item being barely visible on the graph, and all others occupying the same general range (770 to 7,300), although these items are in a slightly different order this time. The main difference between the two graphs is the scale on the X axis is now greatly stretched, indicating that the impact of each item has been magnified.

Comments about benefits

Since quantification was difficult, participants’ quality-based comments appear in the Table 6, below. These comments reveal the participants’ reasoning behind their valuations.

Table 6. Participants’ Comments about Benefits of Usability Testing of Documentation

Summary and conclusions

The costs and benefits of informal usability testing of documentation were studied. The costs were determined with a high degree of confidence, as well as a qualitative list of benefits. Unfortunately, the values for quantitative benefits are not reliable because of the low response from participants. However, impact analysis demonstrates that the quantitative benefits may be enormous, and shows considerable promise for a future study in which participants are trained to track and tabulate the benefits with greater accuracy. This is strong indication that the return on investment for performing informal usability testing of documentation is enormous.

Participants needed for further research

Further study on this topic will of necessity have to include training participants how to track and tabulate the value added by usability testing of their documentation. If you would like to participate in such a study, either as a participant or as a trainer, please contact Elaine Ostrander at the following email address: eostrander@lgc.com or by regular mail at 18102 Bambridge, Houston, Texas 77090.

SIG Home | About the SIG | SIG Activities | Resources | Topics Newsletter |  Conference |  Bookshelf  | Toolkit  Comments or questions? Please send your email to <stcusability at sufficiently.com> © 1998-2004, Society for Technical Communication