Eight Issues to Consider When Developing Metrics for Your Technical Communication Group

Reprinted from the Summer/Fall 2002 Management SIG News, newsletter of the STC Management SIG.

Wondering how you can assess the effectiveness and productivity of your work? Admittedly, it’s not easy and there are no simple approaches. But it can be done.

As you develop a program, consider these issues, which arose from a review of literature on the metrics used to assess the productivity and effectiveness of software engineering, training, marketing communications, and technical communication.

1. Avoid using a single measure or metric to evaluate a product.
Software design and development, training, and marketing communications each track several metrics, each for effectiveness and productivity, and use each to evaluate a different aspect of the work.

Spilka observes that, even within technical communication, “most industry authors advocate a holistic perspective leading to multiple definitions of quality” (210). She adds, “every measure has its unique weaknesses and cannot by itself indicate whether a product or process is high in quality” (212). Furthermore, “uniformly giving primacy of some ingredients of high quality documentation over others can be a risk, because how much these ingredients matter will vary from one context to the next” (211).

2. Before beginning a project, determine how effectiveness and productivity will be evaluated.
Measuring effectiveness and productivity after the fact requires up-front planning before the fact. To be credible, results cannot be measured after the fact without comparing the data to data collected before the project began.

3. Use objectives to assess effectiveness.
Many people define effective technical communication as that which helps users perform intended tasks. To do that, technical communicators must first state what those tasks are, and express them in terms that can they can be observed and measured.

Objectives are statements that express, in observable and measurable terms, what users should be able to do after using a technical communication product. State them before beginning any design work on a project (to make sure that the designs ultimately help users achieve the stated objectives). After publication, assess the extent to which users can perform the intended tasks.

4. Avoid measuring items that do not assess effectiveness or productivity.
Although the use of color, graphics, white space, index entries, navigation aids, and similar devices are often correlated with the effectiveness of a communication product, their presence does not guarantee effectiveness. So the data has no direct utility.

In addition, measuring these design elements is extremely time consuming. It often requires detailed observation of each product. Collecting this data from each page of a printed communication product and each screen of an online communication is tedious and past experience of many companies in collecting it is that the efforts are not sustained.

5. Use financially based measures of productivity.
Productivity is ultimately a measure of return on a financial investment. For example, marketing communicators measure the productivity of their firms by tracking the financial value of their accounts. They assess the effectiveness of their work by the number of sales a particular advertisement, brochure, or catalog generated. Similarly, trainers measure the appropriateness of their budgets by calculating training budgets as a percentage of payroll.

6. Measure intangibles.
Saffir and Turrant observe that “quantitative evaluation is not the ‘be-all and end-all’ but a yardstick” by which organizations can assess their work (212). Although most metrics involve calculation of a hard number, training and marketing communications also include measurements of perceptions and feelings. For example, post-class evaluation of training courses assesses users' emotional reaction and enjoys wide acceptance among training management. Similarly, because they cannot measure financial results, public relations specialists measure perceptions and name recognition, which is the ultimate intent of their work.

7. Review industry surveys done on similar kinds of products. Metrics represent a comparison.
Certainly those taken within an organization help participants track performance among projects. But many organizations want to compare their performance to similar processes performed by other organizations. Industry surveys in training and marketing communications allow professionals to compare some aspects of their work across organizations.

In many fields (not just these), professional organizations like the American Society for Training and Development conduct these surveys. Similarly, publishers serving the trade organization, like IDG (which publishes ComputerWorld and other trade publications), also conduct industry surveys on software development.

8. Remember that while results build perceptions, they do not prove value.
Although practitioners of software design and development, training, and marketing communications each rigorously collect data about their effectiveness and productivity, these practitioners have not shown that the data alone can prove the value of a service. Instead, it is used as part of a larger system of building and maintaining relationships with internal and external sponsors (Robinson & Robinson).

Although these issues do not specifically direct how to assess your effectiveness and productivity, they provide an important foundation that you can use to use to develop methods that work within your organization.

References

Robinson, D. and Robinson, J. (1989.) Training for impact. San Francisco, CA: Jossey Bass.

Saffir, Leonard and John Turrant. (1994.) Power public relations: how to get PR to work for you. NTC business books. Lincolnwood (Chicago), IL.

Spilka, Rachel. (2000.) The issue of quality in professional documentation: how can academia make more of a difference. Technical communication quarterly. 9(2). 207-220.