STC Usability SIG Newsletter
Usability Interface

How Usability and Audit Contribute to Product Design

By Rik Manhaeve

It is almost impossible to do business without using information technology (IT) systems, whether or not they are developed in-house. Evaluating the quality of these systems is critical to an organization’s ability to do business using resources in an optimal way.

The success of any organization derives from being cost effective and optimizing resources. Quality in usable systems is simply good business. Usable systems save effort and allow employees to concentrate on the job at hand. High-quality systems save time and reduce bottlenecks, work-arounds, and support calls.

An audit organization doesn’t measure quality, but it must provide assurance to the Board and senior management that every reasonable action is taken to do "good business." This activity provides a golden opportunity for audit and usability groups to work together to create awareness, to do what has to be done, and to check whether everything is done as it should be. When the collaboration is successful, it leads to what is called "quality in use."

Risks of Low Quality

Creating IT systems is not without risk. The Standish group notes in their 1994 and 2001 reports that 50% of success factors for IT projects are business related. Projects fail either completely or partially more than 70% of the time, and 44% of the most critical factors of failure are business related.

Examples of Risk

Low quality information

Management makes decisions based on the information they get. If systems have a low "quality in use" rating, this information may be too late, inadequate or even incorrect. This has immediate repercussions on the decisions that are based on this information. Moreover, the information about quality may be misleading. Management can have a false feeling of trust and that is even more dangerous.

Waste of investment

The system may not have the needed functionality or may make it almost impossible to follow standard business operating procedures. In these cases users may revert to other ways of working and may totally circumvent the system. The system is not used as intended because it does not work as it should. The cost of development may be wasted.

Loss of productivity

The users do not understand the system, and so they make mistakes but are not helped to correct these mistakes. In this case, the system is not preventing mistakes as it should, so that some users must help others to use the system and are thus distracted from doing their jobs. Other drains on productivity include time spent doing processes twice when the system doesn’t work right, whether the second time uses the new system, or whether users revert to the older method. Productivity does not reach the expected level that is needed to achieve the business goals.

Loss of effectiveness

If results are not achieved in due time, or they are not achieved to their full extent, business goals may not be met effectively. In some cases, such as e-commerce, users may even use the systems of the competition.

Loss of control

If a system has low "quality in use" and the users have the option to not use the system, they will revert to other means to do their tasks. The system may have the appropriate controls but because the system is not used (partially or completely) the business process may go out of control.

Waste of operational cost

Users need training to operate the system; they need help if they don’t understand. Requirements evolve over time, so systems must be adapted to meet the new requirements.

If the system has low "quality in use," users may need more training and more support. Changes are needed faster, not to adapt to new requirements but to make the system workable. All these costs are minimized if the system has good "quality in use."

Unhappy users

When a system does not support them well, users are not able to do their jobs as expected; they make mistakes and lose time. This makes them frustrated and stressed and may even lead to sickness, which further reduces productivity.

Less Prestige for IT

The IT shop is not highly valued by many users, and delivering systems which have low "quality in use" will not help the situation. Users often find they submit comments and demand adjustments but the situation doesn’t improve (even adjustments have low "quality in use"). In the end, users do not submit comments any longer because it does not help—often they then try other ways to achieve their goals, bypassing IT where they can.

Audit Approach to Quality

Auditors use a range of norms and best practices to verify if processes are executed as they should be and to recommend remedial actions if needed.

Major instruments used by Auditors include the following:

The Committee of Sponsoring Organizations of the Treadway Commission (COSO) framework evaluates the goal-directed approach of business processes by looking at the internal control. The definition of COSO states:

Internal control is a process, effectuated by an entity’s board of directors, management and other personnel, designed to provide reasonable assurance regarding the achievement of objectives in the following categories:

• Effectiveness and efficiency of operations
• Reliability of financial reporting
• Compliance with applicable laws and regulations

The effectiveness and efficiency of the operation is directly influenced by the "quality in use" of the IT systems used by the business.

The CobiT framework is directly geared at the way IT operates and has little to do with the way the business uses the IT systems.

The IT Infrastructure Library (ITIL) framework provides descriptive guidance (what to do and why) on IT Service Management.

COSO should look at the performance of the business processes, but most of the time the impact of the IT systems is neglected.

CobiT and ITIL frameworks do not look at the product (the IT system itself), but only at the processes that create IT systems. COSO should look at the performance of the business processes, but most of the time the impact of the IT systems is neglected. These CobiT and ITIL frameworks do not look at the product (the IT system itself), but only at the processes that create IT systems.

Usability Norms and Guidelines for Quality

High "quality in use" is the result of a well structured and managed process. But is the process delivering as expected? Only the "quality in use" of the product will prove it.

ISO 9126 Quality Life Cycle Model (parts 1-4)

ISO 9126 looks at the product itself. The characteristics of the system are investigated during all phases of the whole lifecycle (development and usage), as described here:

Part 1 sets the scene, defines the terms, and gives a whole picture of "quality in use."

Part 2 describes more in detail the quality characteristics of systems during development.

Part 3 describes the characteristics for these systems before their final implementation (the environment is the same as in real usage, but still remains somehow a laboratory setting).

Part 4 gives the characteristics in real usage.

Parts 2, 3, and 4 also define what to do, what to measure, how, and even when and by whom.

ISO/TR 18529—Usability Maturity Model

ISO/TR 18529 describes the processes needed to create good usable systems. The title is "Ergonomics—Ergonomics of human-system interaction—human centered life cycle process descriptions."

The Usability Maturity Model—Processes and Scale (Jonathan Earthy, Lloyd’s Register) states how the capability of a company can be measured and evaluated. Actions can also be defined based on this model.

Joining Forces: Usability and Audit Groups

Looking at the process is good but not enough. To have a complete picture, you need to evaluate the product. To incorporate "quality of use" in the products of IT, you must

• Create awareness and prepare a business case
• Create systems according to an appropriate process
• Evaluate the created systems

Create Awareness

"Quality in use" does not happen without support. Management must be convinced that quality is important, IT must incorporate quality in their development methodology, and the business must perform the evaluation and monitoring. Audit and Usability groups can identify and quantify the risks, as well as develop the business case to present to management.

Create Systems According to an Appropriate Process

Not only must management be aware, but IT must also be committed enough to incorporate the needed tasks in their development approach. In the long run it should not be done as an aside, but rather in the mainstream of their operations. They must acquire the necessary skills and give these activities the necessary emphasis.

Systems must be evaluated during development. Waiting until the product is ready for shipment is not a good way of working because the risks must be identified and mitigated during the development cycle. Auditors must evaluate if the IT shop is acting as it should by inspecting their processes and asking the business for the systems they need to verify their conclusions.

Evaluate the Systems

The business receives the product (the IT system) and must use it within its own activities. Productivity is a major concern for the business. The requirements must be met, but often this is a moving target due to the changes in the business.

Evaluating the system when delivered will not suffice. Continuous evaluation based on changing requirements is needed to be able to maintain the level of "quality in use."

Conclusion

Auditors and Usability groups must join forces. Auditors should use the ISO norms, and not merely concentrate on the processes within IT and the business, but also needs to understand the activities to be able to give relevant recommendations. Looking at the processes is not enough; the product must also be evaluated, and traditionally Audit is not inclined in this direction.

The ISO 9126 standard specifies in great detail what, when, who, and how. The ISO 18529 standard and the Usability Maturity Model state the actions needed to achieve higher "quality in use" and define how to evaluate the capability of the organization to act in this way.

References

CoBit: www.controlit.org.

Committee of sponsoring organizations of the Treadway Commission (COSO): www.coso.org

ISO 9126, Quality Life Cycle Model: www.qa-systems.com/concepts/iso9126.html

IT Infrastructure Library (ITIL): www.itil.org.uk

The Standish Group: www.standishgroup.com/chaos_resources/index.php

Rik has white paper to clarify the connection between Usability and Audit. If you want a copy of that paper, please contact him at hendrik.manhaeve@pandora.be