AJIS vol. 7, no 2 May 2000 ADAPTING THE LOCALES FRAMEWORK FOR HEURISTIC EVALUATION OF GROUPWARE Saul Greenberg', Geraldine Fitzpatrick2, Carl Gutwin3 and Simon Kaplan2'4 'Department of Computer Science, University of Calgary, Alberta, T2N 1N4, Canada saul@cpsc.ucalgary.ca; www.cpsc.ucalgary.ca/-saul 4Department of Computer Science and Electrical Engineering / 2CRC for Distributed Systems Technology, University of Queensland, St. Lucia, QLD 4072, Australia [ger, s.kaplan]@dstc.cdu.au; www.dstc.edu.au 'Department of Computer Science, University of Saskatchewan, Saskatoon, Canada gutwin@cs.usask.ca; www.cs.usask.ca/faculty/gutwin ABSTRACT Heuristic evaluation is a rapid, cheap and effective way for identifying usability problems in single user systems. However, current heuristics do not provide guidance for discovering problems specific to groupware usability. In this paper, we take the Locales Framework and restate it as heuristics appropriate for evaluating groupware. These are: 1) Provide locales; 2) Provide awareness within locales; 3) Allow individual views; 4) Allow people to manage and stay aware of their evolving interactions; and 5) Provide a way to organize and relate locales to one another. To see if these new heuristics are useful in practice, we used them to inspect the interface of Teamwave Workplace, a commercial groupware product. We were successful in identifying the strengths of Teamwave as well as both major and minor interface problems. KEY WORDS: Groupware evaluation, heuristic evaluation, inspection methods, locales framework. INTRODUCTION HCI researchers and practitioners now have a good repertoire of evaluation methodologies that they can use to determine the usability of conventional single user systems. These methodologies range from: those generating quantitative analyses vs. qualitative subjective descriptions; those that are precise but expensive to perform vs. those that are approximate but cheap to do; and those that require user involvement vs. inspection techniques that can be done without user participation. For example, controlled experimentation typically produces objective quantitative results, is precise but expensive, and requires observations of users. In contrast, inspection techniques such as heuristic evaluation and task-centered walkthroughs typically produce qualitative and subjective descriptions of usability problems, are inexact but very cheap to do, and require no user involvement. The point is that evaluators can choose an appropriate methodology to fit the needs and nuances of their situation. In contrast, there is no equivalent set of evaluation methodologies readily available to groupware researchers and practitioners. While some methods have been adapted from other disciplines, these are typically too expensive to use for everyday evaluations, are difficult to learn, or are not appropriate for the questions being asked. As one example, evaluators using ethnographic methods are usually experienced ethnographers, and the studies are almost always expensive in terms of time, logistics and analyses. Similarly, controlled experimentation applied to groupware evaluation is difficult to design and analyze due to the variability inherent in group composition and interaction, the lack of proven metrics that are capable of measuring things of interest in groupware, and the resilience of groups in adapting their behaviors to different controlled situations. The consequence is that groupware is difficult to evaluate. As a result, the CSCW community now includes as part of its research agenda the development of evaluation methodologies for groupware. For example, some researchers are developing a 'discount' version of ethnography called design ethnography, where observations and analyses are specific to design implications rather than to a general understanding of a culture. Other researchers are trying to identify metrics specific to groupware. In this vein, our goal is to see if heuristic evaluation, a cheap but effective evaluation methodology, can be adapted to groupware evaluation. We recognized that existing heuristics, while excellent for evaluating single user systems, do not provide the evaluator with any guidance for discovering problems specific to groupware. Thus our first step in pursuing this goal was to create a new set of heuristics. Fortunately, we felt that an existing theory of group behavior called the Locales Framework was a promising place to begin. In this paper, we present our adaptation of the Locales Framework as heuristics for evaluating groupware. In Section 2, we give a brief summary of both heuristic evaluation and the Locales Framework. In Section 3, we list the groupware heuristics adapted from the Locales Framework. In Section 4, we summarize our experiences using these heuristics to inspect the interface of Teamwave Workplace, a commercial groupware product. We close by outlining the next steps in our research agenda. 102 AJIS vol. 7, no 2 May 2000 BACKGROUND In this section, we briefly review the basic premises of heuristic evaluation and the Locales Framework. Further details are available through the references: both are well documented and well-known to the research community. Heuristic Evaluation Heuristic evaluation (HE) a widely-accepted discount evaluation method for diagnosing potential usability problems in user interfaces (Nielsen 1993, 1994a,b). HE defines a particular interface inspection process (Nielsen and Mack 1994) where several evaluators examine an interface and judge its compliance with recognized usability principles called 'heuristics' (Nielsen, 1994a, p. 26). Non-compliant aspects of the interface are captured as interface bug reports, where evaluators describe the problem, its severity, and perhaps even suggestions of how to fix it. HE is popular with both researchers and industry for a variety of reasons. First, it is low cost: it can usually be done in a few hours, and it does not require any end-user involvement. Second, it is easy to learn: the heuristics are well documented and worked examples are available (e.g., Nielsen 1994a, b). Third, it is becoming part of the standard HCI curriculum (e.g., Greenberg 1996) and thus known to many HCI practitioners. Fourth, it can be used fairly effectively by trained non-usability experts (Nielsen 1994a). Finally, it has a good success rate, where the pooled results of 3-5 evaluators typically identify about 75-80% of all usability problems (Nielsen 1994b). What makes HE particularly powerful is that considerable thought has gone into the selection of heuristics. Starting with a large body of literature of interface guidelines, Nielsen (1994b) examined how well particular guidelines explained usability problems held in a database. The result was a set of about ten recommended heuristics, where the set collectively describes the majority of problems commonly found in today's graphical user interfaces. The catch is that these heuristics are very general, such as "Prevent errors" (Nielsen 1994a). This level of generality means that heuristics arc motherhood statements that serve only to guide the inspection rather than prescribe it: an evaluator must exercise his or her judgement and interpretation skills to uncover particular problems. The Locales Framework The Locales Framework was developed as a principled approach to help people understand the nature of social activity and work, and how a locale (or place) can support these activities (Fitzpatrick 1998; Fitzpatrick, Mansfield, and Kaplan 1996). In this framework, a social world is a group of people with some common purpose, a site for collaboration, and some means to communicate. A locale is the actual site in which a group collaborates, the actual means by which people communicate, and the actual means by which the work is achieved. For example, a team with the goal of designing a software system would form the social world. If the team met in a meeting room, the room and all its artifacts—its visual and auditory communication, its whiteboard, its table, and the paper within it— would form a physical locale. If instead the team met through a groupware system in conjunction with a telephone, then that would form a virtual locale. While a virtual locale may not be actually realized in physical space, it still provides the site and means for the social world to collaborate. More formally, the locales framework comprises five aspects, as summarized here. Further details are in Fitzpatrick, Mansfield, and Kaplan (1996) and the framework is described completely in Chapter 8 of Fitzpatrick (1998). • Locale foundations define a collection of people, resources and artifacts (tools, objects, information) in relation to the central purpose of the social world. • Mutuality considers those interactions within locales that maintain a sense of shared place. • Individual view over multiple locales acknowledges that individuals can be participating in many locales. • Interaction trajectories concern how courses of action evolve over time. • Civic structures concern how interactions fit within a broader communal level. DERIVING GROUPWARE HEURISTICS FROM LOCALES FRAMEWORK Nielsen's analysis that resulted in his recommended heuristics were derived from how well they explained usability problems found in single user environments, but not in groupware. To understand why this makes existing heuristics insufficient for groupware evaluation, we first describe the differences between single user usability and groupware usability. We then present new heuristics based on the Locales Framework. Single-user usability is the degree to which a system is effective, efficient, and pleasant to use, given a certain set of users and tasks (e.g. Shackel 1990). Usability thus emphasizes 'task work': how a person does the domain tasks and activity that produces things like drawings, documents, or models. Clearly, a groupware system must also allow task work to proceed effectively, efficiently, and pleasantly. However, groupware must go beyond task work and support teamwork—the work of working together—in order to be truly usable. Teamwork involves several activities. For example, group members must communicate, organize joint action, provide assistance, coordinate activity, divide labor, and monitor each other's work. Our conception of groupware usability focuses on teamwork, and we define it as the degree to which a groupware system supports the activities of collaboration (Gutwin and Greenberg 1998). 103 AJIS vol. 7, no 2 May 2000 Although teamwork also involves social and affective activities, we limit our definition to those that accomplish the mechanics of collaboration: communication, coordination, planning, monitoring, and assistance. While existing usability heuristics could identify problems in task work, they say nothing about teamwork. Yet developing new heuristics for group work is not straight forward. Nielson had a good corpus of interface development guidelines to work with, but there is no equivalent corpus specific to groupware and teamwork. As an alternative, we decided to use the Locales Framework as the basis for new heuristics. It seemed amenable to this because (and similar to a corpus of guidelines) it is a general rather than specialized framework for understanding fundamental aspects of teamwork, and because it describes a small set of different but inter-dependent perspectives of the characteristics of teamwork, where each could be reconsidered as a heuristic. It is also one of the few theories or frameworks specifically created with groupware in mind. It has also been validated as both a way to understand existing work practices, and to motivate the design of new systems. Thus it is reasonable to expect that it can also be used as a standard against which to evaluate groupware, particularly those that claim to support long-term team activity within some kind of 'space' or 'place'. The Locales Framework was not developed as a usability evaluation method. Rather, it was developed to understand the nature of social practices in the workaday world, and to help inform the design of groupware. Still, converting the framework into heuristics is straightforward. Using the text and descriptions found in Chapters 1 and 8 in Fitzpatrick (1998), we recast them as statements that ask whether a groupware's interface allows certain social phenomena to occur. As with Nielsen's heuristics, these statements are brief reminders to help guide evaluators. Detailed explanation describing the nuances behind each statement are found in Fitzpatrick (1998). Provide centers (locales) that collect people, artifacts and resources in relation to the central purpose of the social world. A locale provides the site, means and resources for a group to pursue team and task work. Locales should be dynamic so they can evolve along with the people, the artifacts, and the purposes that defines them (Harrison and Dourish 1996). Provide awareness (mutuality) within locales that helps people maintain a sense of shared place and that keeps them informed about shared activity. Mutuality includes one person's awareness of others, the artifacts comprising the locale, where things are located, and how things are changing (Gutwin & Greenberg 1999). Allow individual views so one can view a locale or aggregate multiple locales as they relate to one's responsibilities, activities, and interests. A particular person should be able to view locales from his or her particular perspective and in a way that reflects their degree of focus and participation. Allow people to manage and stay aware of their evolving interactions over time. This includes a group's control over past, present and future aspects of routine and non-routine work; how people coordinate and negotiate plans and activities over time; how people leverage past experiences; how breakdowns are noticed and repaired; and how processes are supported. Provide a way to organize and relate locales to one another (civic structures). Locales are rarely independent of one another: people need a way to structure the locales in a meaningful way, to find their way between locales, to create new locales, and to remove old ones. 104 AJIS vol. 7, no 2 May 2000 ji-'lnl x| ', F8e -Edit Room Tack Hep .! ̂ , : Nowr> Teamwave Heuristic Evaluation J Institute for Ihe future Foyer'yererttf Patrick] Draft note) lex 02CHI papei ^udy Louise^ Pde20t) ' 'Pointer: * ! •Other Connected Users Judy Louise (Idle 20s) • Teamwava Heuristic Evaluation1 Jeteroy Patrick. (Idte4