Reader's Questions: Severity Scales
by Chauncey Wilson
Reprinted from Usability Interface, Vol 5, No. 4, April 1999
A reader asks, 'What is a convenient severity scale for classifying usability problems?'
Chauncey Wilson responds:
The usability severity scale should have the same levels as your programming bugs scale. If you have a scale of 1 - 5 with 1 being the most severe problem, the usability scale should also use a 1 - 5 scale. Attributes that should be considered when rating the severity of usability bugs include:
Performance performance is often a primary usability attribute. Performance can be poor (e.g. searching a database takes several minutes to complete) or too good (e.g. autoscrolling can be difficult on a fast machine).
Probability of loss of critical dataAn example would be the wrong default button on the confirmation message 'Do you want to delete this message?'. Choosing 'Yes" for the default button would be very serious because it would cause the loss of the entire database.
Probability of errorWhat is the impact of the error on time, money, reputation (for example, you make the default 'reply to all' and your whole team gets a message meant for only one person).
Violations of standards Some companies have user interface standards. Violation of standards at one company I worked for was a high-level usability bug even if the actual violation was not too severe. Standards are mandatory so violations of standards usually carry a higher penalty than a violation of guidelines.
Impact on profit, revenue, or expensesA high-volume data-entry system should be optimized for the keyboard. Forcing a user to switch from the keyboard to the mouse repeatedly could easily slow input down and increase expenses. For high-volume input, too many keyboard-mouse transitions could be a severe usability problem.
Aesthetics, readability, clutterDoes the user cringe when the screen comes up? Can the user find information in dense screens? This is sometimes a hard attribute to judge, but for Web pages and GUIs this is a serious issue.
Other attributes can affect the severity of a usability problem. Providing some rationale for a severity rating is useful in convincing developers that usability bugs can be just as bad as programming or reliability bugs.
The following severity levels are ones that can be adapted to many bug reporting systems.
Level 1Catastrophic error causing irrevocable loss of data or damage to the hardware or software. The problem could result in large-scale failures that prevent many people from doing their work. Performance is so bad that the system cannot accomplish business goals.
Level 2Severe problem causing possible loss of data. User has no workaround to the problem. Performance is so poor that the system is universally regarded as 'pitiful'.
Level 3Moderate problem causing no permanent loss of data, but wasted time. There is a workaround to the problem. Internal inconsistencies result in increased learning or error rates. An important function or feature does not work as expected.
Level 4Minor but irritating problem. Generally, it causes loss of data but the problem slows users down slightly, minimal violations of guidelines that affect appearance or perception, and mistake that are recoverable.
Level 5 Minimal error. The problem is rare and causes no data loss or major loss of time. Minor cosmetic or consistency issue.
It is important for the Usability Engineer to attend meetings where development and product managers review bugs, decide if the severity is appropriate, and choose which bugs will be fixed. I've been able to convince development and product management to consider some usability bugs as critical bugs. I consider attendance at bug reporting meetings a way to 'train' development and management about usability bugs. It is very helpful if your bug database has an easy way to produce a report on usability bugs.
Jakob Nielsen describes usability severity ratings in his book, Usability Engineering (pages 102 - 104). Nielsen has a useful table that relates the impact of the problem and the proportion of users who will experience the problem to the severity of the usability bug.
One last word of advice: Be judicious in your assignment of severity. Don't make every usability bug a level 1, catastrophic bug or you will get nasty emails from development managers and may suffer a loss of credibility. Good bug hunting!