Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 Beyond information access: Support for complex cognitive activities in public health informatics tools Kamran Sedig 1 , Paul Parsons 1 , Mark Dittmer 1 , Oluwakemi Ola 1 1 Western University, Canada Abstract Public health professionals work with a variety of information sources to carry out their everyday activities. In recent years, interactive computational tools have become deeply embedded in such activities. Unlike the early days of computational tool use, the potential of tools nowadays is not limited to simply providing access to information; rather, they can act as powerful mediators of human-information discourse, enabling rich interaction with public health information. If public health informatics tools are designed and used properly, they can facilitate, enhance, and support the performance of complex cognitive activities that are essential to public health informatics, such as problem solving, forecasting, sense-making, and planning. However, the effective design and evaluation of public health informatics tools requires an understanding of the cognitive and perceptual issues pertaining to how humans work and think with information to perform such activities. This paper draws on research that has examined some of the relevant issues, including interaction design, complex cognition, and visual representations, to offer some human-centered design and evaluation considerations for public health informatics tools. Keywords: public health informatics tools, cognitive science, design, visual representations, interaction Introduction Public health is an information-intensive field [1]. Public health professionals work with a variety of information sources to carry out their everyday activities. For instance, in a recent study of public health systems, Merrill and colleagues [2] report professionals’ top ten tasks by time spent; six of these tasks involve working directly with information (e.g., internet use and data reporting), and the remaining four involve information use mediated by interpersonal communication (e.g., telephone use and meeting with clients). Furthermore, many of the tasks associated with public health risk analysis and clinician-population relationship management involve extensive information collection, analysis, synthesis, and management [3, 4]. In light of the information-intensive nature of public health work, public health researchers and professionals must carefully consider the efficacy of the tools they use to access and work with information. http://ojphi.org/ Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 It is through the access and use of information that the health of populations and communities can be assessed, the causes of disease and injury can be reasoned about, immunization policies can be planned, and decisions regarding building codes and food safety measures can be made. The appropriately termed field of Public Health Informatics (PHI) has grown in recent years to deal with the information needs of public health professionals. In general, informatics focuses on the acquisition, storage, and use of information in a specific setting or domain [5]. As O’Carroll et al. [1] note, it is the principles underlying the respective domains that distinguish various informatics specialty areas from one another. PHI, then, can be broadly considered as the discipline focused on the acquisition, storage, and use of information in areas of public health— health promotion, disease and injury prevention, infectious disease surveillance and reporting, and so on. A necessary realization for any field of informatics is that information has primacy over technology. Technology, although essential, is only a means by which users “make best use of information” [5]. Much of the existing PHI literature focuses on the technological structures that facilitate information access [6, 7, 8, 9]. In such contributions, as well as in review papers that consider multiple PHI tools (e.g., [10, 3]), criteria for evaluation generally consist of meeting basic information access needs [11]. Even the discourse on design principles in the public health literature concerns itself with limited issues such as secure access to varied local, national, and international information sources [12]. Although information access is a prerequisite to information use, the nature of public health informatics work demands that researchers go further and study the patterns of human-information interaction prevalent in the field. Public health informatics work is a form of knowledge work. As such, this work can be characterized in terms of various complex cognitive activities. For example, public health professionals make sense of large datasets; they reason about relationships between demographics, behaviours, and health outcomes; they plan intervention strategies; and they make decisions that affect public policy [13, 14, 15]. In practice, these cognitive activities tend to be both (a) complex, in that they are co-occurring, mutually reinforcing or embedded within each other, and (b) unstructured/open-ended, in that they do not admit a single correct process or solution (for a more elaborate explanation of complex cognitive activities, see [16, 17, 18]). Much of our recent work has been devoted to studying the relationship between such complex cognitive activities and interactive, computer-based tools. One product of this inquiry has been a collection of related frameworks for the design and evaluation of computer-based tools that support complex cognitive activities, elaborated in a series of papers in the fields of human- information interaction, visualization, and visual analytics (see, for example, [16, 19, 20, 21, 22]). This paper draws on elements of these frameworks to offer considerations regarding human-centered design of public health informatics tools. More specifically, we are concerned with public health informatics tools that support, facilitate, and/or enhance complex, unstructured, and/or open-ended cognitive activities. This class of PHI tools would include highly visual tools for epidemiological simulation, analysis, and decision support such as STEM [23], DDSS [24], SOVAT [25], EpiScanGIS [26], MDAST [27], Zeilhofer and colleagues’ component-based tool [28], Epinome [29], and PanViz [30], but exclude simple medical alert notification systems such as those described by Lombardo and colleagues [7] or Gesteland and colleagues [31]. Throughout the paper we refer to this class of tools as PHI tools. http://ojphi.org/ Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 When engaging with any computer-based tool, users 1 meet and interact with information through the visually perceptible interface of the tool. Perception of the current state of the interface is the primary external aid to users’ cognitive activities. Furthermore, users mentally reify information through the visual representations they perceive—that is, from a user’s perspective, the visual representations are the information itself. In this way, the interface is the epistemic locus of any PHI tool, and its design warrants careful consideration. Interface design involves two closely related components: representation and interaction [16]. Representation design involves encoding information in visual forms. For example, consider an epidemiologist using map-based visual representations to reason about incidents of influenza in a region. Two possible encoding schemes of sub-region incident rates are: 1) a colour spectrum, and 2) varying saturation levels of the same colour. In this case, the latter encoding is better suited to support users’ reasoning. A colour spectrum requires users to memorize the mapping of individual colour’s various incident rates, whereas varied saturation naturally expresses a numerical value where higher saturation represents a higher value. This simple example illustrates how different encoding schemes entail different perceptual and cognitive effects that may either enhance or hinder users in accomplishing their goals. Interaction design is concerned with at least the following: 1) what users can and should do with the represented information, 2) what actions should be made available to users to work and think with the represented information, and 3) what reactions should result from users’ actions [16]. Returning to the epidemiology example presented above, let us suppose that individual incidents of influenza are represented as markers on the map. In areas with high incident rates, a dense, difficult-to-understand representation may result. In this case, an interaction that allows filtering out some of the markers could assist the epidemiologist to reason about the situation. Furthermore, a dynamic, adjustable filter that operates on particular sub-regions of the map may be more appropriate than a static, non-adjustable filter that operates on the map globally. This simple example is intended to illustrate that interactive features of a tool may affect how users perform complex cognitive activities and whether their goals are accomplished effectively. If public health informatics researchers are to support professionals using new information visualization tools, they must consider the perceptual and cognitive issues involved in interface design. Researchers interested in the intersection of health informatics and cognitive science have noted that many failures arise due to a lack of understanding of, and consideration for, human issues in design—that is, lack of understanding of human-centric design. For instance, Zhang [32] notes that most failures “are not due to flawed technology, but rather due to the lack of systematic considerations of human and other non-technology issues in the design and implementation processes.” Such non-technology, human-centered design issues are the primary focus of this paper. The structure of this paper is as follows. First, we describe some typical complex cognitive activities that users of PHI tools perform. Second, we describe the nature of public health information spaces. Third, we present some considerations for how this information may be represented. Fourth, we discuss interaction design for PHI tools. Fifth, we discuss interactivity and its role in the effectiveness of tools. Sixth, we present a brief scenario in which ideas from 1 In this paper, ‘users’ is a blanket term that can refer to public health researchers, practitioners, scientists, decision makers, policy makers, and so on. http://ojphi.org/ Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 previous sections are applied to the design of a PHI tool. Finally, we present some concluding remarks. Complex Cognitive Activities in Public Health Informatics Consider a public health professional using a PHI tool to investigate the factors that contribute to child obesity in a region. After the user gains access to the pertinent information, he or she must (a) plan which factors to investigate and in what order, (b) make sense of information from a variety of sources, (c) analyze and reason about interaction effects between various factors that may vary over time and across the region, and finally (d) decide upon intervention strategies and policy recommendations in light of new findings. In other words, the user must perform a number of different complex cognitive activities. As the investigation unfolds, the user may move fluidly between these activities, perform tasks that serve more than one activity at once, or perform one activity as a part of completing another. For example, the user may periodically return to planning as new goals or tasks come to light. Building statistical models may assist the user in making sense of data, analyzing correlations, and reasoning about cause and effect. Finally, reasoning and analysis may also be embedded within decision-making, insofar as the purpose of the former is to provide evidence for the latter. Each of these activities will potentially involve several tasks and sub-tasks that are completed through interactions with the PHI tool at the level of the interface [16]. Effective design of a specific PHI tool, therefore, depends upon an accurate model of how the tool should support the complex cognitive activities at hand. Such a model may emerge from experience in the field, empirical studies, and/or participatory design. In many situations, patterns of cognitive behaviour across users are diverse, and flexibility is required to accommodate user characteristics, needs, and preferences. Designers must offer appropriate visual representations and interaction abilities for users to carry out meaningful discourse with information in a way that best supports their pertinent tasks and activities. Public Health Information Spaces Before approaching issues of visual representation and interaction, researchers and designers must consider what information is relevant to their users. In this paper, we regard information as an external and objective entity as described in [33], not as a subjective construct (i.e., knowledge in one’s head) or as a process (i.e., the act of informing) as described in [34]). We refer to the sum total of information relevant to the user as the information space—for a more thorough discussion of the concept, see [16, 20]. Public health information spaces are often heterogeneous [35]. Users access and interact with different types of information from a variety of sources to perform complex cognitive activities. For instance, consider a public health state official dealing with a local influenza outbreak. This official may begin by analyzing regional hospital records or state vaccine supply information. As the official’s task unfolds, he or she may review school-based surveillance systems and recent federal recommendations on handling outbreaks before selecting an appropriate course of action. A decision support tool for this official must be designed with due consideration for the scope and content of this information space. http://ojphi.org/ Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 Of course, researchers and designers may not have a priori knowledge of the full extent of information that is relevant to their users. As the above example illustrates, information spaces are context-dependent, and can change over time. This challenge can be overcome by employing flexible methods of information access, or by considering how a particular PHI tool may be situated alongside other PHI tools at the user’s disposal. In the above example, a designer may provide explicit access to each information source within a single PHI tool, or integrate their PHI tool with other tools that provide such access. In any case, a thorough characterization of the user’s information space should preface visual representation and interaction design. Visual Representations of Information Once the user’s information space is well understood, the designer must determine how information will be represented. Information only becomes accessible to users when it is given form through the visually-perceptible interface of a tool. 2 Visual representations play a variety of roles in supporting users to accomplish their tasks; they can anchor and structure thought processes, provide a medium for offloading memory and mental operations, reduce the cognitive effort required by changing the nature of the task, and provide explicit encoding of information for collaboration [36, 37, 38]. The utility of visual representations, however, depends on the designer’s ability to select representations that are appropriate to user tasks and activities. Visual representations combine and integrate low-level visual marks, such as lines, dots, and other shapes, into more complex structural forms. These forms, along with visual variables that pre-attentively influence perception, such as colour, orientation, texture, and size, represent and encode information items (e.g., entities, properties, relationships, processes) that exist within an information space. It is possible that different representations of the same information can have very different cognitive effects [39]. Furthermore, different representational forms can significantly impact how complex cognitive activities are performed (e.g., [36, 40]). Consider a public health professional studying access to healthcare facilities. This professional uses a tool that displays information about local hospitals, including (a) whether the hospital contains an emergency room or urgent care center, (b) the number of beds available in various departments, and (c) the distribution of patient wait times. Figures 1 and 2 illustrate two visual representations of this information. Figure 1 is a map-based visual representation with repeated symbols to represent beds in different departments and a histogram indicating patient wait-time distribution. Figure 2 contains a list of hospitals with a bar graph representation of bed counts and a time-varying plot of individual patient wait-times. Both figures encode hospitals with emergency rooms in red, and hospitals with urgent care centers in blue. Overall, the map is probably more useful because it facilitates tasks related to the spatial distribution of hospitals. The histogram emphasizes the overall distribution of patient wait times, whereas the time- varying plot emphasizes peak times; the most appropriate representation is dependent on the context and the tasks that must be performed to meet the needs and goals of the user. 2 Although information may be made accessible to any one of the senses, this paper is concerned with only visual representations. The terms ‘representation’ and ‘visual representation’ are used interchangeably throughout the paper. http://ojphi.org/ Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 Figure 1. Map-based representation of hospital facilities Figure 2. List-based representation of hospital facilities Note that none of the above visual representations encodes the entire information space. To assess access to healthcare facilities, the user may wish to see population density, areas with high concentration of vulnerable individuals, transit routes, and walk-in clinic locations, among other information. All of this information is unencoded—it remains latent within the information space and is inaccessible to the user. Encoding all of this additional information, however, would likely result in clutter and confusion. The tension between unencoded information and information overload is problematic. Public health professionals are inundated with massive amounts of information. As a result, practitioners are beginning to demand that information be presented in a compact form that can be readily absorbed [41, 42, 43]. The size of the public health information space means that only a small fraction of it can be encoded into representations at any given time. The only way for the user to access latent, unencoded information is through interaction. Interaction Print-like, static representations, though beneficial, do not allow the user to actively manipulate representations in ways that may reveal latent information [44], thereby requiring users to bear the brunt of the information-processing load. Representations that are interactive can help bridge the gap between the internal mental representations of the user and external visual representations of the tool [16, 19]. Such dynamic, interactive representations offer users http://ojphi.org/ Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 flexibility by supporting convergent and divergent thinking, and accommodate evolutionary and iterative patterns of thought [45, 46, 47]. Patel and Kushniruk point out that some computer-based tools in the field of health, when implemented as “short term fixes”, tend to disrupt (rather than enhance) users’ complex cognitive activities [48]. A deeper understanding of interaction and its benefits is crucial for creating tools that support, rather than hinder, cognition [44, 45, 46, 49]. Calls for a more comprehensive understanding of the cognitive issues surrounding interaction have been ongoing for the past decade, and span various fields including health informatics, information visualization, human-computer interaction, visual analytics, and information science (e.g., see [32, 44, 45, 46, 50, 51, 52]). Much of our recent work has been devoted to characterizing various facets of interaction. Interaction is a complex phenomenon, and it is difficult to capture all of its rich characteristics. To mitigate this problem, PHI tool designers may find it helpful to think of user activity in terms of several levels of interaction. In our work, we have categorized interaction into four levels: (1) complex cognitive activities as they are presented above, (2) tasks and sub-tasks such as locating or categorizing, (3) individual interactions such as filtering or transforming, and (4) user- interface events such as clicking or swiping [16, 20]. PHI researchers and designers may approach these levels from the top down, viewing low-level patterns as embedded within high- level patterns, or from the bottom up, viewing high-level patterns as emerging from the combination of low-level patterns. In either case, identifying the prevalent combinations of activities, tasks, and interactions that users will perform allows designers to combine interaction techniques that are appropriate to a series of anticipated user contexts. Even a cursory survey of the literature in interaction design, information visualization, visual analytics, and related disciplines uncovers an overwhelming abundance of interaction techniques. For PHI researchers and practitioners, interaction techniques that have a potentially infinite variety of implementations and are scattered throughout numerous disciplines are of little help in supporting the systematic design and evaluation of PHI tools. Furthermore, as new technologies are developed, old techniques may fall out of use as techniques more appropriate to the new technology arise. What is needed, then, is the development of frameworks and taxonomies that identify and explicate underlying patterns that are platform-, technique-, and technology- independent. Interaction Patterns One aspect of interaction to which we have devoted considerable research effort is the identification and explication of fundamental patterns that are used to interact with information during the performance of complex cognitive activities (also referred to as action patterns [16]). Such research stems in part from the observation that interaction techniques are generally operationalized at the level of individual interactions, that is, user-action/system-reaction pairs. Since there are potentially infinite interaction techniques, interaction patterns allow designers to situate techniques within general categories according to their function. Some of the patterns from [16] are arranging—changing the ordering of visual representations, scoping— dynamically working forwards and backwards to view compositional development and growth of information, selecting—focusing on or choosing particular visual representations, translating— converting visual representations into alternative informationally- or conceptually-equivalent http://ojphi.org/ Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 forms, and transforming—changing the geometric form of visual representations. Each pattern supports the performance of complex cognitive activities in different ways, and there are innumerable interaction techniques that fall under each pattern. Categorizing techniques according to the patterns that they express has numerous benefits. In designing PHI tools, considering a manageable variety of interaction patterns and their cognitive effects allows designers to systematize their interaction design process and justify their design decisions in a clear manner. Additionally, in evaluating PHI tools, interaction patterns offer a common vocabulary for discussing interaction at a low level that is resilient to technological change. Consider the PHI exploration tool illustrated in Figures 3 - 8 using [53]. The user is exploring the relationship between low birth weight and various external factors. In Figure 3, the user has an unordered list of average low birth-weight prevalence. By clicking a menu option, the user arranges the items in descending order (as shown in Figure 4). This example is one way of implementing an arranging action pattern; of course, there are many other ways, but all of them achieve the same purpose for the user. That is, the user changes the ordering of the represented information to facilitate tasks such as ranking, classifying, and identifying information items. Figures 5 - 8 illustrate how a sequence of other action patterns—selecting items (Figure 5), translating from one visual form to another (Figure 6), and scoping through changes over time (Figures 7 and 8)—reveals previously undetectable patterns in the information space. For a more comprehensive discussion of these and other interaction patterns, see [16]. Figure 3. Initial state of bar graph representation Figure 4. Ordered bar graph representation http://ojphi.org/ Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 Figure 5. Bar graph representation with selection Figure 6. Bar graph translated into bubble chart Figure 7. Bubble chart encoding correlation with female head of household (%) and infant mortality rate Figure 8. Bubble chart scoped from 2002 to 2005 Interactivity In addition to interaction considerations, PHI tool designers must also consider the quality of interaction, or interactivity, that emerges through use of their tool. This consideration is important because research has shown that factors that affect the quality of interaction have significant cognitive effects (e.g., see [54]). One of our lines of research identifies facets of interaction that affect interactivity. We have recently devised a framework that outlines a number of such facets and discusses how they influence the quality of interaction, and ultimately, the performance of complex cognitive activities (see [19]). Just as interaction can be characterized at many different levels, so can interactivity. At the level of individual interactions, a number of elements can give structure to an interaction (see [19]). For example, one such element, called presence, is illustrated in Figures 9 and 10. Presence is concerned with the existence and advertisement of an action. In other words, it is about the cue or signal from the interface used to prompt the user or to advertise the existence of an interaction. This structural element can be operationalized in one of two ways: explicitly or implicitly. If presence of an action is explicit, the availability, existence, or provision of the interaction is clearly advertised by the tool. When presence is implicit, the interaction exists, but http://ojphi.org/ Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 its availability is either not easily perceptible by the user, or it is not visible at the level of the interface. In this case, the user must have previous knowledge of the existence of the interaction. Figures 9 and 10 illustrate the two operational forms of presence. The figures depict two implementations of the drilling interaction pattern which, in this case, causes region-specific statistics to be displayed. In Figure 9, the presence of the interaction is operationalized in an implicit manner, and the user must hover the mouse cursor over a region to discover that doing so displays more information. In Figure 10, the presence of the interaction is operationalized in an explicit manner, advertised as the “info tool” in a toolbar on the left. The obvious design trade-off in this case is weighing the cost of screen space for the toolbar against the risk that the user will not discover unadvertised functionality. However, there is a more subtle cognitive effect at play. The implicit implementation has no “drilling mode” alongside other modes of interaction; hovering the cursor over a region always drills the region, just as clicking a region or clicking a statistic always performs the same action. In the explicit case, the toolbar forces users to think about their actions in terms of the currently selected modality; clicking a region with the “info tool” will not have the same effect as clicking with the “zoom tool”. As a result, the implicit implementation is more predictable whereas the explicit implementation is more flexible. The framework in [19] identifies 11 other elements that collectively give structure to an interaction. Similar to presence, each element has different operational forms, each affecting cognitive processes in different ways. If researchers and designers are aware of these elements of interactivity and have an idea of how they affect the quality of interaction, PHI tools can be designed in a manner that more effectively facilitates the performance of complex cognitive activities. Figure 9. Drilling by district requires mouse hover and is unadvertised: implicit presence Figure 10. Drilling with the info tool is advertised in a toolbar: explicit presence The quality of human-information interaction at a higher level can also be examined according to [19]. At this level, the concern is not the structure of an individual interaction, but the combination and sequencing of several interactions and how they affect the performance of tasks and activities. The framework in [19] has identified a number of factors that influence interactivity at a high level, one of which is flexibility. Flexibility is concerned with the degree to http://ojphi.org/ Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 which the user can adjust the properties of the interface to suit his/her needs, preferences, characteristics, and goals. Flexibility is particularly relevant to PHI because PHI users’ goals, characteristics, and needs are diverse [55, 56]. As a result, researchers have called for development of tools that cater to this diversity (e.g., [55, 57]). One way of supporting a diverse audience is to incorporate greater flexibility into PHI tools. One facet of flexibility is concerned with whether a tool allows users to adjust the settings of different ontological properties of visual representations. Parsons and Sedig [21] have identified a number of ontological properties of visual representations, the settings of which influence cognitive and perceptual processes during the performance of complex cognitive activities. For instance, one ontological property of all visual representations is density—the degree to which information items are encoded compactly in the visual representation. Depending on the task and the user, different settings (i.e., degrees) of density are most appropriate. Therefore, to design PHI tools that are human-centered, users should be given the ability to adjust such settings to suit their needs and preferences. Other properties of visual representations identified in [21] include appearance—the aesthetic features (e.g., color and texture) by which information items are encoded, dynamism—the degree to which encoded information items exhibit movement, scope— the degree to which the growth and development of information items are encoded, and type— the form of a visual representation in which information items are encoded (see [21] for a full discussion of these and other properties). An awareness of the ontological properties of visual representations and how their settings can and should be made adjustable is vital to any systematic endeavor concerned with the use of visual representations in complex cognitive activities. For example, consider a public health practitioner using a GIS tool to analyze cases of Chlamydia Trachomatis, such as in [8]. Depending on the user’s needs, he or she may wish to adjust the type of representation to better suit a particular task. If the user is presented with a bar graph, for instance, such a representational form may not be ideal for performing tasks such as identifying the spatial distribution of infection cases. By allowing the user to adjust the type of representation to a cartogram, for example, the task being performed can become more tractable. A user may need to adjust the type of representation many times to facilitate different tasks during the performance of an overall activity. Each of the properties identified in [21] have been shown to affect perceptual and cognitive processes and, therefore, giving users the ability to adjust such properties can enhance the quality of interaction by contributing to the flexibility of a tool. Concrete examples of this facet of flexibility are drawn from open access tools dealing with PHI datasets and are illustrated in Figures 11 - 14. The Google Public Data Explorer tool (Figures 11 and 12) provides the user with the ability to adjust the type settings of visual representations, so that the same underlying dataset can take on the following forms: line graph, bar graph, geographic map, or bubble chart [53]. The Spatio-Temporal Epidemiological Modeller (STEM) visualizes the spread of disease over time (Figures 13 and 14) [23]. Often times, it is the case that users adjust the settings of multiple properties with the same interaction—a phenomenon that can be deliberately designed when designers understand how adjusting the settings of different properties can facilitate a user’s tasks. In Figures 13 and 14, for instance, the user is adjusting both the dynamism and scope settings—by increasing or decreasing the degree of movement and the degree to which the growth and development of the information items are encoded. Figures 13 and 14 show the spread of influenza through Great Britain on Day 32 and 54, respectively, of a simulated outbreak. Adjusting the settings of each property has distinct benefits for performing http://ojphi.org/ Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 certain tasks (see [21]), and combining these adjustability options can provide better support for tasks related to the spread of the disease. Figure 11. Adjusting type settings (1): a bar graph representation Figure 12. Adjusting type settings (2): an equivalent map-based representation Figure 13. Adjusting dynamism and scope settings (1): representations are static Figure 14. Adjusting dynamism and scope settings (2): representations exhibit movement to depict the development and spread of disease Design and evaluation of PHI tools can be rendered more systematic through awareness of the many elements and factors that influence interactivity (see [19] for a more complete discussion). Since the discussed frameworks do not deal with interaction and interactivity at the implementation level, they can promote both systematicity and creativity in the design process. For instance, if designers consider the aforementioned factor of flexibility, allowing users to adjust the settings of ontological properties of visual representations frees designers from preoccupation with selecting the single “right” representation. Users are instead empowered to tailor representations to suit their contextual and cognitive needs, while designers are free to direct their energy toward creating novel interaction techniques and combinations and sequences of interactions. http://ojphi.org/ Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 The frameworks discussed above have a number of benefits for PHI researchers and practitioners: they offer a common vocabulary for discussing interaction and interactivity; they are platform-, tool-, and technology-independent; they facilitate systematic comparison and contrast of PHI tools; and, finally, they promote systematic design with an understanding of how numerous design considerations affect the performance of complex cognitive activities. Scenario: Design of a PHI tool This section illustrates how the aforementioned frameworks can be integrated to assist with the design and evaluation of PHI tools 3 using the scenario mentioned at the beginning of the second section of this paper, in which a public health professional investigates the factors contributing to child obesity in a region, in this case the region being the USA. With the frameworks discussed above, designers can structure their thinking in a comprehensive and holistic manner. Design decisions can be made systematically, with an understanding of how they influence the performance of complex cognitive activities. For instance, as previously mentioned, such a scenario involves planning, sense-making, analytical reasoning, and decision-making. If designers and evaluators are not aware of the characteristics of such activities, and how to best support them with interactive visual representations, PHI tools cannot be designed and evaluated effectively. Using the ideas discussed in this paper, however, designers can think systematically about such issues as the characteristics of the information space with which they are concerned; how to encode different aspects of the information space with visual representations; how representations emphasize different aspects of information and act as lenses through which users perceive the information space; what actions should be made available to the users to work and think with the represented information; how the different ways of operationalizing interactions influence cognitive processes; which properties of the representations should have adjustable settings to best support the users’ tasks; how interactions should be combined and linked together to facilitate the performance of tasks; and, how the interplay of all of these considerations ultimately influences how complex cognitive activities are performed. Some of these considerations will be briefly discussed in the context of the scenario described above. We ground our discussion of these concepts with a feasible design that is presented through a functional description and is accompanied by a series of user interface mock-ups 4 . Our intention is to illustrate, in broad strokes, one potential realization of these concepts that is suitable to the scenario. It is important to note that the mock-ups should not be mistaken for a polished tool ready for use by public health professionals. In this scenario, the information space with which users are concerned includes health datasets that may originate from multiple sources. These data include categorical and quantitative variables such as age group, sex, socioeconomic status, prevalence of disease, environmental factors, and so on. Some of these data may be instances applied to individuals, but most are aggregated at various levels—e.g., by district, by state, or by nation. The information space also includes past and current policy decisions, information about existing programs to combat 3 This section is concerned with both design and evaluation of PHI tools. For ease of reading the terms ‘designer’ and ‘evaluator’ are used interchangeably. 4 Mock-ups, here, refer to rough sketches illustrating the user interface layout. http://ojphi.org/ Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 obesity, and obesity-related research. This information is most readily represented in the form of digitized documents describing policy, programs, and research. During analysis, the user will deal with a multitude of variables. In addition, he or she will be interested in statistical models derived from these variables, from linear correlations between prevalence of obesity and a single other variable, to complex models involving a multitude of variables. These can be represented using many different forms, such as maps, scatter plots, and bar graphs, each of which emphasize different aspects of the information space and have different perceptual and cognitive effects. For example, Figure 15 shows a variety of variables represented via colour saturation and bar graphs, while regions with a moderate or strong significant correlation with an obesity measure are hatched or cross-hatched, respectively. Such encodings have been shown to have cognitive effects at the pre-attentive level of perceptual processing. The user can pan the map-based representation using the scroll wheel of the mouse; by default, the map pans vertically, and depressing the shift key modifies the scroll wheel behaviour to pan horizontally. Figure 15. A typical view of the scenario tool containing a large map-based visual representation (top left), a series of notes containing lists and links (top right), and a listing of variables and statistics along with a set of saved representations (bottom) Through a variables-and-statistics dialogue (see Figure 16) users can adjust several properties of representations. For example, users can adjust density by selecting the level of aggregation; adjust appearance by selecting features such as colour and shape; and adjust type by creating graph, plot, or map-based representations. When the user interacts with the list of statistics (see right side of the modal dialogue in Figure 16), in addition to the above-mentioned controls for http://ojphi.org/ Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 controlling the statistic’s representation, the user is presented with a form for selecting multiple dependent variables and a type of statistical model from which the statistic is derived before being represented on the map. The user may either alter an existing statistical model by selecting it in the list or create a new model by clicking the “+” entry in the list (see rightmost list in modal window in Figure 16). The variables and statistics dialogue can be opened by double-clicking either (a) a variable or statistic name, or (b) a representation of a variable or statistic. Through this dialogue, new visual representations can be added to the view, and existing representations can be translated from one form to another. Figure 16. A variable-and-statistics dialogue for modifying representations displayed by the tool The density and complexity properties can also be adjusted through two discrete slider controls along the right of the map view (see Figure 15). The top control adjusts the zoom level of the map itself. The bottom control contains two adjustable controls on one scale. One control is a discrete marker, similar to the one used for zoom level, that dictates the level of granularity at which the user may select a region for drilling, such as by district or by state. The other control is a two-ended range that dictates the levels of aggregate granularity that are represented. For example, the user may wish to select and drill by state while viewing map regions, graphs, and plots defined for both district and state levels of granularity. Zooming and filtering with controls is an example of operationalizing interactions with an indirect form of focus—one of the structural elements from [19] that affects interactivity. Even though the user acts upon the control, what he or she is trying to achieve is a change in the map-based visual representation— i.e., though the user’s focus is on the control, the user indirectly affects the map. The drilling interaction brings into view statistical representations that cannot be overlaid directly onto the map. Multiple-selection of regions allows the user to view one or more regions http://ojphi.org/ Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 at once to make sense of relationships between variables and between regions. In this way, there is strong complementarity between the selecting and drilling interactions. If the user wishes to make a direct comparison, plots of multiple regions can be superimposed by dragging one plot onto the other in one continuous motion. When the user completes the action, the two plots animate to merge together into a single plot; an example of the end result is shown in the rightmost scatter plot in Figure 17. This use of continuous action/reaction flow—another element from [19] that affects interactivity—provides users with an intuitive action pattern for combining plots (through dragging) while aiding comprehension of the final result (through a smooth transition). A cylindrical icon represents each data source used to generate statistics. This representation can be further drilled (not shown here) to expose important details such as sample size, year of data collection, related research goals, and related publications. Figure 17. A multi-region drilling example When the user discovers a new pattern or correlation, the visualization state can be saved using the “+V” button (see Figures 15 and 17). Ad-hoc visual compositions can also be constructed by clicking and dragging one visual representation, such as a map region or bar graph, onto the “+V” button. In both cases, the button expands to create room for the user to enter a title for the view. When the user presses ‘enter’, the view is added to the stack of views next to the “+V” button. Once a new view has been stored, other ad-hoc elements can be added to it by clicking and dragging them onto the view’s representation on the stack. The dragging-and-naming method of storing a view is an example of an interaction with composite interaction granularity in which multiple steps performed by the user complete a single logical interaction. This example illustrates how composite interactions allow the user to specify certain parameters that affect the outcome of the interaction, in this case, the name assigned to the stored view. http://ojphi.org/ Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 To facilitate transitions between planning, decision making, and analytical reasoning, the user is provided with a note-taking component that supports linking to visual representations (in red), external documents (in blue), and other notes (in yellow); examples are shown in the rightmost panel in Figure 15. Views that are linked to a note are also annotated with a note icon that can be used to open notes about a view as the user works with the representations (see yellow icons in bottom-right of drilling dialogue in Figure 17). This is another example of complementarity between two patterns—linking and annotating. The note-taking component also supports task lists that automatically instantiate a task summary when a task is checked off (see last item in task list in Figure 15). The task summary workflow allows the user to describe his/her findings after completing a task and to link to relevant documents and visual representations. Clearly documenting plans and outcomes in this way assists users to be systematic in their analysis and locate relevant information in the case of preparing a report. Table 1 summarizes the interaction patterns from [16] that can help support thinking about design and evaluation of the tool in this scenario. Although brief, this section has attempted to illustrate how the ideas presented in this paper can facilitate the systematic design and evaluation of PHI tools. Only a few considerations were examined here; however, if the frameworks discussed above are considered together, the result is a comprehensive support structure that facilitates coherent and holistic thinking about how numerous complex cognitive activities can be performed using interactive PHI tools. http://ojphi.org/ Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 Table 1. Detailed interaction design matrix for scenario tool Interaction Pattern Functional Description Annotating Add personal meta-information to linked visual representations Assigning Bind characteristics to visual representations in variable/statistics modal dialogue Comparing When the drilling dialogue shows data from multiple regions, click one visual representation and drag it onto the corresponding representation for another region to identify their degree of similarity Composing / Decomposing Drag visual representations onto an existing representation icon to compose new ad-hoc representations; drag representations outside the tool to break them apart Drilling Bring out and encode latent information from map regions and data sources by double-clicking or selecting and pressing the enter key Filtering Show or hide elements of visual representations with discrete slider Linking / Unlinking Establish a relationship between visual representations by selecting and dragging them into note, or with right-click menu or keyboard shortcut Selecting Click visual representations or icons; click with the shift key depressed for multiple selection Storing / Retrieving Put aside or bring back visual representations by clicking the “+V” button Translating Depress the ctrl key and click a visual representation to open the variable/statistics window; the user may choose to convert representations into alternative informationally- or conceptually- equivalent forms Summary For public health professionals to efficiently ensure and promote the health of the general populace, they must engage with a variety of information sources to perform their everyday activities. Focusing on only information access is insufficient, however, if PHI tools are to become powerful enough to support users in the complex cognitive activities associated with PHI practice. What is needed is a clear understanding of how users of PHI tools engage in a dynamic discourse with public health information in order to assess the health of populations and communities, reason about causal chains that lead to disease, plan immunization policies, and perform other public-health-related complex cognitive activities. To develop such an understanding, the features of this human-information discourse that influence the performance of such activities must be identified, characterized, and explicated. Such features include the encoding and representation of items from information spaces into visual forms; the ontological properties of visual representations and how their settings influence cognitive and perceptual processes; the actions that should be made available to users to work and think with the represented information; the different ways in which such actions and their subsequent reactions should be operationalized; and how all of these considerations ultimately http://ojphi.org/ Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 combine to influence problem solving, sense-making, analytical reasoning, and other complex cognitive activities. Such an understanding necessarily involves the integration of research from numerous disciplines, including public health informatics, computer science, cognitive science, information visualization, and human-computer interaction design. Furthermore, to ensure that relevant research is not scattered and inconsistent, theoretical models, frameworks, and taxonomies that deal with the fundamentals of the aforementioned issues must be developed and made available to researchers, designers, evaluators, analysts, and users. Over the past decade, we have been conducting research to develop a number of inter-related frameworks that bring order and structure to the general area of human-information interaction in complex cognitive activities. We believe that such research can be of great benefit to the area of PHI. In this paper, we have identified some of the extant research needs for PHI tools, presented some fundamental concepts that must be understood if PHI tools are to effectively support complex cognitive activities, discussed a number of the considerations from our frameworks that have implications for the design and evaluation of PHI tools, and provided a scenario to demonstrate how the integration of such considerations facilitates deliberate and methodical design and evaluation. As PHI moves beyond mere information access and becomes concerned with enabling rich discourse with information in order to support complex cognitive activities, the considerations discussed in this paper become critical to the effective design and evaluation of PHI tools. As research in this area is still new and emerging, it is hoped that this paper will make a valuable contribution to the PHI literature and will stimulate much future work in this area. Acknowledgements The authors would like to thank the Natural Sciences and Engineering Research Council of Canada for their financial support. Corresponding Author Kamran Sedig Western University, Canada Email: sedig@uwo.ca References 1. O’Carroll PW, Yasnoff WA, Ward ME, Ripp LH, Martin EL, eds. Public Health Informatics and Information Systems. Springer-Verlag; 2003. 2. Merrill J, Keeling J, Gebbie K. Toward standardized, comparable public health systems data: A taxonomic description of essential public health work. Health Services Research. Wiley Online Library; 2009;44(5p2):1818–41. http://ojphi.org/ mailto:sedig@uwo.ca Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 3. Nykiforuk CIJ, Flaman LM. 2011. Geographic Information Systems (GIS) for Health Promotion and Public Health: A Review. Health Promot Pract. 12(1), 63-73. http:// dx.doi.org/10.1177/1524839909334624 4. Goodman KW. 2010. Ethics, Information Technology, and Public Health: New Challenges for the Clinician-Patient Relationship. The Journal of Law, Medicine & Ethics. Wiley Online Library. 38(1), 58-63. 5. Hersh W. 2009. A stimulus to define informatics and health information technology. BMC Med Inform Decis Mak. 9(1), 24. http://dx.doi.org/10.1186/1472-6947-9-24 6. Braithwaite G, Haggard LM. 2006. Utah’s IBIS-PH: an innovative user interface solution for Webbased data query systems. J Public Health Manag Pract. 12(2), 146-54. http:// dx.doi.org/10.1097/00124784-200603000-00006 7. Lombardo JS, Garrett N, Loschen W, Seagraves R, Nichols B, et al. 2009. An Informatics Solution for Informing Care Delivery of Immediate Public Health Risks to Their Patients. Online J Public Health Inform. 1(1). http://dx.doi.org/10.5210/ojphi.v1i1.2750 8. Savel TG, Bronstein A, Duck W, Rhodes MB, Lee B, et al. 2010. Development and Evaluation of GIS-Based Chlamydia Trachomatis Intervention Policy in Illinois. Online J Public Health Inform. 2(1). 9. Studnicki J, Fisher JW, Eichelberger C, Bridger C, Angelon-Gaetz K, et al. 2010. CATCH: Advancing Public Health Analytics. Online J Public Health Inform. 2(3). http://dx.doi.org/10.5210/ ojphi.v2i3.3348 10. Reeder BP, Turner A, Demiris G. 2010. Use of Technology to Support Information Needs for Continuity of Operations Planning in Public Health: A Systematic Review. Online J Public Health Inform. 2(1). http://dx.doi.org/10.5210/ojphi.v2i1.2855 11. Dixon BE, Grannis SJ. 2011. Why “What Data Are Necessary for This Project?” and Other Basic Questions are Important to Address in Public Health Informatics Practice and Research. Online J Public Health Inform. 3(3). http://dx.doi.org/10.5210/ojphi.v3i3.3792 12. Neame R. 2012. Design Principles in the Development of (Public) Health Information Infrastructures. Online J Public Health Inform. 4(1). http://dx.doi.org/10.5210/ojphi.v4i1.4022 13. Homer JB, Hirsch GB. 2006. System dynamics for modelling public health: Background and Opportunities. Am J Public Health. 96(3), 452-58. http://dx.doi.org/10.2105/AJPH.2005.062059 14. Leischow SJ, Best A, Trochim WM, Clark PI, Gallagher RS, et al. 2008. Systems thinking to improve the public’s health. Am J Prev Med. 35(2), 196-203. http://dx.doi.org/10.1016/ j.amepre.2008.05.014 15. Trochim WM, Cabrera DA, Milstein B, Gallagher RS, Leischow SJ. 2006. Practical challenges of systems thinking and modelling in public health. Am J Public Health. 96(3), 538-46. http:// dx.doi.org/10.2105/AJPH.2005.066001 16. Sedig K, Parsons P. Interaction design for cognitive activity support tools: A pattern-based approach. AIS Transactions on Human-Computer Interaction. (Accepted). 17. Knauff M, Wolf AG. 2010. Complex cognition: the science of human reasoning, problem-solving, and decision-making. Cognitive Processing. Springer. 11(2), 99-102. http://dx.doi.org/10.1007/ s10339-010-0362-z 18. Sternberg RJ, Ben-Zeev T. Complex Cognition: The Psychology of Human Thought. NY: Oxford University Press; 2001. 19. Sedig K, Parsons P, Dittmer M, Haworth R. Human-centered interactivity of visualization tools: Micro- and macro-level considerations. In: Huang T (ed.) Human Centric Visualization: Theories, Methodologies and Case Studies. Springer; in press. 20. Sedig K, Parsons P, Babanski A. 2012. Towards a characterization of interactivity in visual analytics. Journal of Multimedia Processing Technologies. Special Issue on Theory and Application of Visual Analytics. 3(1), 12-28. http://ojphi.org/ http://dx.doi.org/10.1177/1524839909334624 http://dx.doi.org/10.1177/1524839909334624 http://dx.doi.org/10.1186/1472-6947-9-24 http://dx.doi.org/10.1097/00124784-200603000-00006 http://dx.doi.org/10.1097/00124784-200603000-00006 http://dx.doi.org/10.5210/ojphi.v1i1.2750 http://dx.doi.org/10.5210/ http://dx.doi.org/10.5210/ojphi.v2i1.2855 http://dx.doi.org/10.5210/ojphi.v3i3.3792 http://dx.doi.org/10.5210/ojphi.v4i1.4022 http://dx.doi.org/10.2105/AJPH.2005.062059 http://dx.doi.org/10.1016/ http://dx.doi.org/10.2105/AJPH.2005.066001 http://dx.doi.org/10.2105/AJPH.2005.066001 http://dx.doi.org/10.1007/ Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 21. Parsons P, Sedig K. Properties of visual representations: Improving the quality of humaninformation interaction in complex cognitive activities [JASIST]. J Am Soc Inf Sci Technol. (Under review). 22. Parsons P, Sedig K. On the utility of common visualizations for performing cognitive activities. In: T. Huang and W. Huang, editors. Innovative Approaches of Data Visualization and Visual Analytics. In press. 23. Ford DA, Kaufman JH, Eiron I. 2006. An extensible spatial and temporal epidemiological modelling system. Int J Health Geogr. 5(1), 4. http://dx.doi.org/10.1186/1476-072X-5-4 24. Lozano-Fuentes S, Elizondo-Quiroga D, Farfan-Ale JA, Loroño-Pino MA, Garcia-Rejon J, et al. 2008. Use of Google EarthTM to strengthen public health capacity and facilitate management of vector-borne diseases in resource-poor environments. Bulletin of the World Health Organization. SciELO Public Health. 86(9), 718-25. 25. Parmanto B, Paramita MV, Sugiantara W, Pramana G, Scotch M, et al. 2008. Spatial and multidimensional visualization of Indonesia’s village health statistics. International Journal of Health Geographics. BioMed Central Ltd. 7(1), 30. 26. Reinhardt M, Elias J, Albert J, Frosch M, Harmsen D, et al. 2008. EpiScanGIS: an online geographic surveillance system for meningococcal disease. Int J Health Geogr. 7(1). http:// dx.doi.org/10.1186/1476-072X-7-33 27. Kramer RA, Dickinson KL, Anderson RM, Fowler VG, Miranda ML, Mutero CM, et al. Using decision analysis to improve malaria control policy making. Health policy. Elsevier; 2009;92(2):133–40. 28. Zeilhofer P, Neto PSA, Maja WY, Vecchiato DA. 2009. A web-based, component-oriented application for spatial modelling of habitat suitability of mosquito vectors. International Journal of Digital Earth. Taylor & Francis. 2(4), 327-42. http://dx.doi.org/10.1080/17538940902887324 29. Livnat Y, Gesteland P, Benuzillo J, Pettey W, Bolton D, et al. Epinome-a novel workbench for epidemic investigation and analysis of search strategies in public health practice. AMIA Annual Symposium Proceedings. 2010. p. 647. 30. Afzal S, Maciejewski R, Ebert DS. Visual analytics decision support environment for epidemic modeling and response evaluation. Visual Analytics Science and Technology (VAST), 2011 IEEE Conference. 2011. p. 191–200. 31. Gesteland PH, Samore MH, Pavia AT, Srivastava R, Korgenski K, et al. Informing the front line about common respiratory viral epidemics. AMIA Annual Symposium Proceedings. 2007. p. 274. 32. Zhang J. Human-centered computing in health information systems. Part 1: Analysis and design. Journal of Biomedical Informatics. San Diego, USA: Elsevier Science; 2005;38(1):1–3. 33. Marchionini G. 2010. Information Concepts: From Books to Cyberspace Identities. Synthesis Lectures on Information Concepts, Retrieval, and Services. 2(1), 1-105. http://dx.doi.org/10.2200/ S00306ED1V01Y201010ICR016 34. Buckland MK. Information as thing. Journal of the American Society for Information Science. John Wiley & Sons; 1991;42(5):351–60. 35. Rambo N. 1998. Information resources for public health practice. Journal of Urban Health: Bulletin of the New York Academy of Medicine. 75(4), 807-25. http://dx.doi.org/10.1007/ BF02344510 36. Zhang J, Norman D. 1994. Representations in Distributed Cognitive Tasks. Cogn Sci. 18(1), 87-122. http://dx.doi.org/10.1207/s15516709cog1801_3 http://ojphi.org/ http://dx.doi.org/10.1186/1476-072X-5-4 http://dx.doi.org/10.1186/1476-072X-7-33 http://dx.doi.org/10.1186/1476-072X-7-33 http://dx.doi.org/10.1080/17538940902887324 http://dx.doi.org/10.2200/ http://dx.doi.org/10.1007/ http://dx.doi.org/10.1207/s15516709cog1801_3 Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 38. Kirsh D. 2010. Thinking with external representations. AI & Society. Springer London. 25(4), 441-54. 39. Peterson D. Forms of Representation. Oxford, UK: Intellect Books; 1996. 40. Larkin J, Simon H. 1987. Why a Diagram is (Sometimes). Worth Ten Thousand Words. Cognitive Science. Elsevier. 11(1), 65-100. 41. Rambo N. 2000. Information Needs and Uses of the Public Health Workforce -- Washington, 1997-1998. Morbidity and Mortality Weekly Report. 49(06), 118-20. 42. LaPelle NR, Luckmann R, Simpson EH, Martin ER. 2006. Identifying strategies to improve access to credible and relevant information for public health professionals: a qualitative study. BMC Public Health. 6(1), 89. http://dx.doi.org/10.1186/1471-2458-6-89 43. Higgins JW, Strange K, Scarr J, Pennock M, Barr V, Yew A, et al. “It’s a Feel. That’s What a Lot of Our Evidence Would Consist of”: Public Health Practitioners’ Perspectives on Evidence. Evaluation & the Health Professions. SAGE Publications; 2011;34(3):278–96. 44. Sedig K, Rowhani S, Morey J, Liang H. 2003. Application of information visualization techniques to the design of a mathematical mindtool: A usability study. Journal of Information Visualization. 2(3), 142-60. http://dx.doi.org/10.1057/palgrave.ivs.9500047 45. Thomas JJ, Cook KA, eds. Illuminating the Path: The Research and Development Agenda for Visual Analytics. Los Alamitos, CA, United States: IEEE Computer Society; 2005. 46. Sedig K, Sumner M. Characterizing interaction with visual mathematical representations. International Journal of Computers for Mathematical Learning. Springer Netherlands; 2006;11(1):1–55. 47. Spence R. Information Visualization: Design for Interaction. 2nd ed. Essex, UK: Pearson; 2007. 48. Patel VL, Kushniruk AW. Interface design for health care environments: the role of cognitive science. Proceedings of the AMIA Symposium. American Medical Informatics Association; 1998;29–37. 49. Yi JS, Kang YA, Stasko J, Jacko J. 2007. Toward a Deeper Understanding of the Role of Interaction in Information Visualization. IEEE Transactions on Visualization and Computer Graphics. IEEE Computer Society. 13(6), 1224-31. 50. Johnson CM, Johnson TR, Zhang J. 2005. A user-centered framework for redesigning health care interfaces. J Biomed Inform. 38(1), 75-87. http://dx.doi.org/10.1016/j.jbi.2004.11.005 51. Marchionini G. 2008. Human-information interaction research and development. Libr Inf Sci Res. 30, 165-74. http://dx.doi.org/10.1016/j.lisr.2008.07.001 52. Arias-Hernandez R, Green TM, Fisher B. 2012. From cognitive amplifiers to cognitive prostheses: Understandings of the material basis of cognition in visual analytics. Interdiscip Sci Rev. 37(1), 4-18. http://dx.doi.org/10.1179/0308018812Z.0000000001 53. Google Public Data Explorer. Google; 2012. Available from: http://www.google.com/ publicdata. 54. Liang H-N, Parsons PC, Wu H-C, Sedig K. 2010. An exploratory study of interactivity in visualization tools: “Flow” of interaction. J Interact Learn Res. 21(1), 5-45. 55. Rambo N, Zenan JS, Alpi KM, Burroughs CM, Cahn MA, et al. 2001. Public Health Outreach Forum: lessons learned. Bull Med Libr Assoc. 89(4), 403-06. 56. O’Carroll PW, Cahn MA, Auston I, Selden CR. 1998. Information needs in public health and health policy: results of recent studies. Journal of Urban Health: Bulletin of the New York Academy of Medicine. 75(4), 785-93. http://dx.doi.org/10.1007/BF02344508 http://ojphi.org/ http://dx.doi.org/10.1186/1471-2458-6-89 http://dx.doi.org/10.1057/palgrave.ivs.9500047 http://dx.doi.org/10.1016/j.jbi.2004.11.005 http://dx.doi.org/10.1016/j.lisr.2008.07.001 http://dx.doi.org/10.1179/0308018812Z.0000000001 http://www.google.com/ http://dx.doi.org/10.1007/BF02344508 Beyond information access: Support for complex cognitive activities in public health informatics tools Online Journal of Public Health Informatics * ISSN 1947-2579 * http://ojphi.org * Vol.4, No. 3, 2012 57. Turner AM, Liddy ED, Bradley J, Wheatley JA. 2005. Modeling public health interventions for improved access to the gray literature. J Med Libr Assoc. 93(4), 487-94. http://ojphi.org/