This article was originally published in the October 2007 issue (Vol 13, No. 1)

Originally presented at a meeting of the Usability Professionals Association chapter of Washington, D.C. on May 24, 2007

STC UUX Community Newsletter
Usability Interface

Innovations in Card Sorting

Reviewed By David Dick

Whether you are designing a system to organize content such as a toolbar, web site or help files, you do it yourself or you can consult actual users and learn how they perceive the content and structure. Usability specialists use card sorting to learn how users think and how information can be organized in a way that makes sense to them. Card sorting is new to me, and I am interested to learn how I can apply it to my work.

On 24 May, the UPA Chapter of Washington D.C. organized a presentation about card sorting given by Justin Dopke and Celeste Lyn Paul.

Introduction to card sorting

The meeting began with an introduction to card sorting given by Justin Dopke, a usability specialist at the Social Security Administration. Creating a product that has a logical information structure is critical to the success of the product. A good structure helps users find information and accomplish their tasks with ease. C ard sorting is one method that can help us understand how users think the information and navigation should be within a product.

Justin explained that participants are given an assortment of index cards, each labeled with the title of content or functionality, and arrange them into groups (categories) that best makes sense to them. The facilitator records the results, and shuffles the cards for the next participant. Card sorting typically involves testing one to nine people who represent diverse user groups, and any number of cards.

There are three types of card sorting:

• Open card sorting, performed early in the design process, to determine patterns in how users group information.
• Closed card sorting is performed to validate and confirm design decisions
• Combination of open and closed card sorting

Justin explained that closed card sorting would be used to redesign a web site. A combination of Open and Closed Card Sorting would be performed to prioritize information and validation. When sorting is done, the results are recorded on a spreadsheet and analyzed to determine patterns of categories and incorporate results into the design. He did not go into the details about how to prepare for a card sort, the methods and software available to analyze results and present the information; because they can be a topic all its own.

Innovations in card sorting

The next presenter was Celeste Lyn Paul, a Senior Interaction Architect at User-Centered Design, Inc. In 2006, The Information Architecture Institute awarded Celeste a grant for her project on "A Delphi Approach To Card Sorting".

According to Celeste, traditional card sorting is not as low cost as proponents would like us to think. From her experience, getting valuable results open card sorting involves at least 20 participants. More participants means more days a testing, fees to pay for facilities and facilitator hours, and the costs of repeated work. Card sorting is mentally exhausting for participants and difficult to analyze results. Is there a faster and better way to conduct card sorting? Yes, there is a better way.

What is the Delphi Method?

The Delphi method was developed at the Rand Corporation in the 1950's and 1960's to forecast the impact of technology on warfare. It is a systematic interactive forecasting method based on independent inputs of selected experts. The Delphi method recognizes the value of expert opinion, experience and intuition.

Celeste adapted the Delphi method to card sorting and called it Modified Delphi Method. She explained how she applied the "modified" Delphi method to card sorting.

The Modified Delphi method to card sorting

Participants are "experts" of a product, the target audience of a future product and represent the primary user group(s). The first participant creates the initial structure; a second participant (and subsequent participants) comments on the previous participant's structure and makes changes. This iteration continues until all participants have arranged the cards and there are no more (significant) changes.

Celeste explained that the first participant (the seed) creates the initial structure. There are several ways to create the seed: (1) an Information Architect creates the initial structure, (2) an expert assists the seed with the initial structure, (3) a single seed works alone, (4) multiple seeds work together. Participants work with the previous participant's structure, and they can change the organization of the structure however they want (small changes, significant changes or ambiguous changes). The organization of the cards evolves until there are no more changes.

The benefits for adopting Delphi is that it offers a single information structure rather than many, achieves insight from only 8 to 10 participants, it is a familiar qualitative methodology, lower study costs, and lower participant (cognitive costs).

Case Study

Celeste summarized a study (thesis) that she conducted at the University of Baltimore School of Law. The participants included staff, law professionals and students. The study involved sorting 90 cards of high-level website topics, using the open card sort and the Delphi method. She asked participants to organize the cards into groups that made sense to them.

• The Open Card method generated greater than fifty percent agreement with 19 of the 90 cards. Also, 8 of 11 final categories represented.
• The Modified Delphi Method generated greater than fifty percent agreement with 66 of the 90 cards. Also, 9 of 10 final categories represented.

You can receive a copy of her University of Baltimore thesis by writing to Celeste at cpaul@user-centereddesign.com.

Final Thoughts

Like any good presentation, the Celeste and Justin covered a lot of information but time did not allow them to go into details about recruiting volunteers and the methods to analyze and interpret results. The audience was eager to share their experiences in conducting card sorting, and ask Justin and Celeste their advice on how to do better next time. I believe that card sorting is a skill that Technical Writers can put to good use.

The following are resources about card sorting that are available from the STC (Society for Technical Communication) website:

Kaufman, J. Card Sorting: An Inexpensive and Practical Usability Technique http://stc.org/intercom/PDFs/2006/200611_17-19.pdf

Topics in Usability: Card Sorting lists a variety of articles and instructions, tools and commentary. http://www.stcsig.org/usability/topics/cardsorting.html

Results of global online sort for world usability day 2006. http://www.stcsig.org/usability/newsletter/0701-card-sort.html

The following are resources recommended by Celeste:

Lamantia, J. Analyzing Card Sort Results with a Spreadsheet Template. Boxes and Arrows. August 26, 2003. http://www.boxesandarrows.com/view/analyzing_card_sort_results_with_a_spreadsheet_template

Maurer, D. and Warfel, T. Card sorting: a definitive guide. Boxes and Arrows. April 7, 2004. http://www.boxesandarrows.com/view/card_sorting_a_definitive_guide.

Paul, C.L. Resources for Modified-Delphi Card Sorting. http://www.obso1337.org/hci/delphi.

Surowieki, J. The Wisdom of Crowds: Why the Many are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. Brown, 2004.

Tullis, T. and Wood, L. How Many Users Are Enough for a Card-Sorting Study? Proceedings of UPA 2004.