DocQment--Quality SIG Quarterly Newsletter

Front page || Quality SIG Home || STC Home

Click here for printer-friendly version.

Quality Special lnterest Group Quarterly Newsletter
Metrics: What We Measure and Why

Managing a group of technical writers on a large project is, at best, like what that anonymous source described as “herding cats.”

And then there is management. Your boss waltzes into your office and asks you to produce “some metrics” for your part of the project. She wants to develop a presentation for her management that will show how well the project is going. (By the way, “she” is not a technical communicator, and this is the first time that the Tech Pubs Group has fallen into her group.)

Your first reaction is to panic. Then your logical, technical communicator’s mind takes over, and you realize that what you need is more information. Where do you start?

The first things to ask yourself are:

What is the story we want to tell?

What is the story that management wants to hear?

Who is management (director of IT, finance, a product manager)?

That brings us to the basic tenets of technical communication, the place where we ask what the purpose of the information is, and who the audience will be.

Let’s begin with two key project indicators: schedule and budget. Management will want to know if you are meeting schedule and are in budget, and, if not, why not.

For some of my larger projects, I have 75 to 125 manuals, some of them multi-volume. My schedules (Primavera reports) and my budget sheets (Excel spreadsheets) are multi-page, often printed on tabloid-sized paper—too much paper and too much information to suit my upper management. What I do to accommodate them is to distill the key information into pie charts or graphs. Areas highlighted as outstanding successes and key problem areas I call out, providing explanations for the successes and challenges. Using such visuals (the charts or graphs) helps to focus management on the areas where I need help or where I want to recognize and reward my staff.

A metric unique to my environment, and one that my project management wants, is Earned Value Ratio (EVR). EVR is a metric that shows expended labor hours against percentage of work completed compared to the total budget. EVR reflects my progress against the budget. If the percent is positive, management wants to know how much money I can return to the project. If the percent is negative, I present a plan on how I can mitigate the loss and bring the plan back into line with the budget.

In addition to metrics for project management, upper management may want to know what Return on Investment (ROI) your part of the project contributes to the bottom line of the Profit and Loss Statement (P&L). If your manual is part of the product, it is easier to calculate ROI. If development of technical information is not part of the capitalized product, you will have to work with your finance people to determine its actual cost. Try to get numbers from your customer support people to show how your documentation will help to reduce support calls or help answer questions.

But what if we need to know if our project (manual/website/online help) is meeting the customer’s needs. Does our company have a Customer Relationship Management (CRM) program? Is there data coming out of the CRM program that we can use to identify areas where we are succeeding in getting the right information to the customer at the time they need it? Is there data that identifies holes in the information we are providing? This is the information that will tell us why the customer wants our product and what they plan to do with it.

If our company does usability testing, can we get our manual included with the usability testing of the product? How does this information tally? Can we show improvement from one version to the next? As with the CRM data, usability testing can show an aspect of the customer’s profile: how the customer uses our product and our information.

Customer support has vital information about how robust our product is. They know the holes in the product and know what the customer is actually doing with the product. Sometimes customers do things with products that they were never designed to do. Sometimes there are undocumented product features. How do these phenomena reflect on our documentation and our metrics?

All of these areas provide data that helps rate and rank employee performance.

However, page counts, sometimes assumed to be a useful metric, are actually a very small indicator of performance. Relying on page count, how would I reward the writer who hones and refines information to reduce the total page count? If, however, I have information from my customers on the quality of content and ease of use of our manual, I can use that information in an employee’s performance review, as well as my management reports, rather than page counts.

Collecting data over time is the best measurement. If you have standard points that you monitor, you can compare projects. You can track changes and discover trends.

“She” has come back into your office. Now you are prepared to ask such questions as:

Who is going to be in the meeting where you are making your presentation?

Do you know some of the questions that you are going to be asked?

How long is your presentation and how much of that time is dedicated to Tech Pubs?

What is the story that you want to tell about this project?

What is the format of your presentation?

References

CRM articles on www.eweek.com

Dumas, Joseph S. and Janice C. Redish. A Practical Guide to Usability Testing. ABLEX Publishing Corporation: Norwood, NJ, 1993.

Hackos, JoAnn. Managing Your Documentation Projects. John Wiley & Sons, Inc.: New York, 1994.

Mead, Jay. “Measuring the Value Added by Technical Documentation.” Technical Communication, Journal of the Society for Technical Communication, August 1998.

Olve, Nils-Göran, Jan Roy, and Magnus Wetter. Performance Drivers: A Practical Guide to Using the Balanced Scorecard. John Wiley & Sons: New York, NY, 1999.

Skelton, T. M. “Managing the Development of Information Products: An Experiential Learning Strategy for Product Developers.” Technical Communication, Journal of the Society for Technical Communication, February 2002.

Spyridakis, Jan H. “Guidelines for Authoring Comprehensible Web Pages and Evaluating Their Success.” Technical Communication, Journal of the Society for Technical Communication, August 2000.

van der Meij, Hans. “Examining the Relationship Between Quality Writing and Quality Reading." Technical Communication, Journal of the Society for Technical Communication, May 2000.

Wick, Corey. “Knowledge Management and Leadership Opportunities for Technical Communicators.” Technical Communication, Journal of the Society for Technical Communication, November 2000.

About the Author: Deirdre Murr, fellow of the STC, is manager of technical publications for Walt Disney Imagineering, Inc. She supervises 17 writers working on over 200 projects with an operating and project budget over $6M. During her 20+ years in the technical communications field, she has spent 9 years managing technical writing departments. The majority of her work has been in the software industry.

Back to top

	Metrics: What We Measure and Why
	Managing a group of technical writers on a large project is, at best, like what that anonymous source described as “herding cats.” And then there is management. Your boss waltzes into your office and asks you to produce “some metrics” for your part of the project. She wants to develop a presentation for her management that will show how well the project is going. (By the way, “she” is not a technical communicator, and this is the first time that the Tech Pubs Group has fallen into her group.) Your first reaction is to panic. Then your logical, technical communicator’s mind takes over, and you realize that what you need is more information. Where do you start? The first things to ask yourself are: What is the story we want to tell? What is the story that management wants to hear? Who is management (director of IT, finance, a product manager)? That brings us to the basic tenets of technical communication, the place where we ask what the purpose of the information is, and who the audience will be. Let’s begin with two key project indicators: schedule and budget. Management will want to know if you are meeting schedule and are in budget, and, if not, why not. For some of my larger projects, I have 75 to 125 manuals, some of them multi-volume. My schedules (Primavera reports) and my budget sheets (Excel spreadsheets) are multi-page, often printed on tabloid-sized paper—too much paper and too much information to suit my upper management. What I do to accommodate them is to distill the key information into pie charts or graphs. Areas highlighted as outstanding successes and key problem areas I call out, providing explanations for the successes and challenges. Using such visuals (the charts or graphs) helps to focus management on the areas where I need help or where I want to recognize and reward my staff. A metric unique to my environment, and one that my project management wants, is Earned Value Ratio (EVR). EVR is a metric that shows expended labor hours against percentage of work completed compared to the total budget. EVR reflects my progress against the budget. If the percent is positive, management wants to know how much money I can return to the project. If the percent is negative, I present a plan on how I can mitigate the loss and bring the plan back into line with the budget. In addition to metrics for project management, upper management may want to know what Return on Investment (ROI) your part of the project contributes to the bottom line of the Profit and Loss Statement (P&L). If your manual is part of the product, it is easier to calculate ROI. If development of technical information is not part of the capitalized product, you will have to work with your finance people to determine its actual cost. Try to get numbers from your customer support people to show how your documentation will help to reduce support calls or help answer questions. But what if we need to know if our project (manual/website/online help) is meeting the customer’s needs. Does our company have a Customer Relationship Management (CRM) program? Is there data coming out of the CRM program that we can use to identify areas where we are succeeding in getting the right information to the customer at the time they need it? Is there data that identifies holes in the information we are providing? This is the information that will tell us why the customer wants our product and what they plan to do with it. If our company does usability testing, can we get our manual included with the usability testing of the product? How does this information tally? Can we show improvement from one version to the next? As with the CRM data, usability testing can show an aspect of the customer’s profile: how the customer uses our product and our information. Customer support has vital information about how robust our product is. They know the holes in the product and know what the customer is actually doing with the product. Sometimes customers do things with products that they were never designed to do. Sometimes there are undocumented product features. How do these phenomena reflect on our documentation and our metrics? All of these areas provide data that helps rate and rank employee performance. However, page counts, sometimes assumed to be a useful metric, are actually a very small indicator of performance. Relying on page count, how would I reward the writer who hones and refines information to reduce the total page count? If, however, I have information from my customers on the quality of content and ease of use of our manual, I can use that information in an employee’s performance review, as well as my management reports, rather than page counts. Collecting data over time is the best measurement. If you have standard points that you monitor, you can compare projects. You can track changes and discover trends. “She” has come back into your office. Now you are prepared to ask such questions as: Who is going to be in the meeting where you are making your presentation? Do you know some of the questions that you are going to be asked? How long is your presentation and how much of that time is dedicated to Tech Pubs? What is the story that you want to tell about this project? What is the format of your presentation?
References	CRM articles on www.eweek.com Dumas, Joseph S. and Janice C. Redish. A Practical Guide to Usability Testing. ABLEX Publishing Corporation: Norwood, NJ, 1993. Hackos, JoAnn. Managing Your Documentation Projects. John Wiley & Sons, Inc.: New York, 1994. Mead, Jay. “Measuring the Value Added by Technical Documentation.” Technical Communication, Journal of the Society for Technical Communication, August 1998. Olve, Nils-Göran, Jan Roy, and Magnus Wetter. Performance Drivers: A Practical Guide to Using the Balanced Scorecard. John Wiley & Sons: New York, NY, 1999. Skelton, T. M. “Managing the Development of Information Products: An Experiential Learning Strategy for Product Developers.” Technical Communication, Journal of the Society for Technical Communication, February 2002. Spyridakis, Jan H. “Guidelines for Authoring Comprehensible Web Pages and Evaluating Their Success.” Technical Communication, Journal of the Society for Technical Communication, August 2000. van der Meij, Hans. “Examining the Relationship Between Quality Writing and Quality Reading." Technical Communication, Journal of the Society for Technical Communication, May 2000. Wick, Corey. “Knowledge Management and Leadership Opportunities for Technical Communicators.” Technical Communication, Journal of the Society for Technical Communication, November 2000.
	About the Author: Deirdre Murr, fellow of the STC, is manager of technical publications for Walt Disney Imagineering, Inc. She supervises 17 writers working on over 200 projects with an operating and project budget over $6M. During her 20+ years in the technical communications field, she has spent 9 years managing technical writing departments. The majority of her work has been in the software industry. Back to top