june06b.indd Brenda Reeb, John D’Ignazio, John Law, and Michael Visser Federated search observed in the context of student writing Taking steps towards improving user experience At the risk of stating the obvious, information technologies in academic libraries grow increasingly complex and interdependent. It’s worth the repetition to underline a countervail- ing force that librarians and vendors provide in taming this complexity for the benefit of patrons. Federated search, a relatively new technology, is a tool built with this goal in mind as it allows librarians to aggregate online content at will, providing a single search box that allows people to retrieve items from multiple content sources with one entry. Implementing federated search technol- ogy creates a new interdependence among three stakeholders: federated search software vendors, content vendors, and library Web designers. With an interest in exploring the impact of next-generation federated search functionality on people’s use of the library, members of each stakeholder group met at the University of Rochester in New York for two days in January 2005 to conduct a series of student interviews.1 The group included staff from the University of Rochester Librar- ies; Endeavor Information Systems, a feder- ated search software vendor; and ProQuest Information and Learning, a content provider. A fourth member of the group, from the Association for Research Libraries, brought experience in user-centered research and design. The group conducted nine interviews with undergraduate test subjects who had recently written a research paper using the contextual inquiry model. The user is the expert Contextual inquiry is an interview method used in contextual design, a methodology used in software and hardware design.2 Im- portant characteristics of contextual inquiry are master/apprentice role-playing, interview questions that are narrowly focused on work tasks, and conducting interviews in the work- place. During an interview, the interviewer assumes the role of an apprentice to the test subject, who teaches mastery in using the technology to perform a task. The roles serve as a device for the interviewer to create a neutral environment for the test subject to demonstrate work habits with a minimum of performance evaluation. Task-specific questions asked in the moment encourage the user to speak and act concretely. Con- textual interview techniques steer the user away from vague summaries of what they wish they could do. Through performance and demonstration of tasks, the test subject recalls tacit knowledge that does not come to mind when talking about work without this context. Exactly what happened during an interview? The project team divided into two groups to increase the number of interview sessions possible over two days. To qualify for par- Brenda Reeb is coordinator of the University of Rochester Libraries’ Usability Team, e-mail: breeb@library.rochester. edu; John D’Ignazio is a doctoral student at Syracuse University’s School of Information Studies, e-mail: jadignaz@syr.edu; John Law does strategic planning and defines new features for various technologies at ProQuest Information and Learning, e-mail: john. law@proquest.com; Michael Visser is product manager for dissertations and theses at ProQuest Information and Learning, e-mail: jmichaelvisser@yahoo.com © 2006 Brenda Reeb, John D’Ignazio, John Law, Michael Visser 352C&RL News June 2006 mailto:jmichaelvisser@yahoo.com mailto:law@proquest.com mailto:jadignaz@syr.edu mailto:breeb@library.rochester ticipation, a test subject had to be an under- graduate and must have written a paper using any library resource in the previous or current semester. Test subjects were selected based on a fi rst-come, first-serve response to an e-mail solicitation and received a $25 gift certifi cate to Amazon.com for their participation.3 Sessions were recorded using Morae event recording software by TechSmith Corp. This software makes it easy to create clips of the interviews that include audio and video with the concur- rent desktop navigation of the user.4 Each interview began by explaining the project and asking for the participant’s con- sent to be recorded. Next, the interviewer invited the test subject to show how he or she gathered material to complete his or her most recent paper-writing assignment. Notice the emphasis on “show” instead of “tell,” in keeping with the contextual inquiry method that emphasizes the person’s experience with his or her work. During this demonstration, the interviewer maintained a conversational rapport, while asking repeatedly for the stu- dent to describe what he or she was doing and why. When the test subject retrieved results from any source, the interviewer shifted to questions to reveal the student’s evaluation process. Sophisticatedly literal Taken as a whole, the interviews gave in- sight into the behavior of undergraduate test subjects who had a variety of information needs. The students’ writing topics included Thomas Chatterton, finch hormones, moral philosophical theories, Shirley Temple’s biography, and ecoterrorism. The most compelling observation from the interviews was the participants’ combination of effort to master their topic with literal and relatively naïve use of the library information systems. Through body language and verbal com- munication, the test subjects demonstrated intense effort to relate the results of their searches with what they wanted to express in their writing and what they learned in class. Clearly the test subjects were engaged in their work. In concert with engagement came a literal interpretation of research methods and search results, such as the way they employed previ- ous bibliographic instruction. One test subject commented, “I learned about ProQuest in my CAS class [freshman writing class, attended over one year ago] and my paper [today] was on Chatterton, so ProQuest is a good database for literature topics.” The test subject returned to ProQuest Academic as a good “literature” database because she had learned about it in a less sophisticated and somewhat tangential course on writing. In this case, as in others witnessed during the sessions, trusted re- sources, whether appropriate for the research project or not, win over a “search all” feder- ated view. The same test subject evaluated her search results by whether the title or ab- stract contained “her words.” Her paper was about Thomas Chatterton’s impersonation of Chaucer, and she scanned all metadata for these four words: suicide, death, imperson­ ation, and Chaucer. The Library of Congress subject heading “Chatterton, Thomas, 1752- 1770 Criticism and interpretation” represents an entirely different approach to accessing possible relevant information. Another test subject who became an avid user of NoodleBib, a citation manage- ment tool from NoodleTools, Inc., in high school, continued to log in to his high school NoodleBib account from Rochester, and never inquired if similar tools existed at the univer- sity.5 He continued to use what had worked for him in the past, avoiding the investment of time to find out if Rochester’s environment would better support his work. Poor design causes pain Test subjects exhibited impatience and irrita- tion in the face of poor design. When research, already feared as a time-consuming process, became more so because the user interface was unhelpful, our participants let us know. For example, comments about the federated search implementation were that it lacked clarity or forced the user to complete many steps to get what they needed. The latter complaint occurred most memorably regarding the open URL June 2006 353 C&RL News http:Amazon.com linking paths, where one test subject blurted, “Can’t you cut the middle man?” in response to the SFX menu screen. Lack of descriptive metadata, such as sum- maries or abstracts, impedes resource selection. Even when it was clear that clicking the title would reveal full text, test subjects used, or expressed preference for, an abstract to avoid reading the entire item during their selection process. A few test subjects used index terms to gauge the relevance of an item for their topic. Context adds value Context provided by a search box or list of databases adds value and trust. A majority of test participants perceived resources pre- sented as course-related as better than the same databases listed on an A-Z database list. In several cases when the test subjects recognized specific products that he or she had used successfully in the past, those were preferred over the federated search box. “Fac- ulty recommended,” either noted on a course page or verbally stated in class, had much more meaning than “search all.” Resources discovered “in context” in these ways seemed more trusted and credible to the students. Otherwise there was little perceived distinc- tion between resources, and we recorded students’ perception that all databases are the same. For example, one test subject stated firmly that all neuroscience databases are the same—each one is current and contains thousands of articles. At a literal level this is a true statement. At a higher level there are important content and search interface distinctions among Ovid’s Medline, PubMed, Biological Sciences, and Science Citation Index, all of which were available to him on the neuroscience databases page.6 The habituated user Test subjects frequently created their own digital workspaces, using common desktop resources to speed accessing and assimilating information for their paper. Sometimes this was compensatory behavior for functionality not found in the native database, but often it was a convenience-based habit. Test subjects used desktop or browser functions such as the “Find and Replace” feature (Control – F) to find search terms in a result set, ignoring native database functionality to narrow search results. “Save” and “Save As,” Select All, and basic cut/paste features common in the desk- top environment were used more frequently than native database mark/e-mail/export features to capture content. After capturing content, some of the test subjects pasted it into Microsoft Word documents or a blank e-mail message that they mailed to themselves. Some test subjects used the browser navigational his- tory to mark their activity despite the problems of replicating their path on different machines. Bibliographic citations, URLs, blocks of text from articles or Web sites are examples of content saved in these seemingly ephemeral digital workspaces. While use of desktop or browser functions was widespread, we had many examples of idiosyncratic behavior, which included one test subject who habitually kept Word open in the background to use the spell check feature in validating her search queries. Nonlibrary content is part of the research landscape Taking us along for the ride, some test sub- jects did not stay in the “safe” information environment provided by the library, but used Internet resources such as Google and Amazon.com. One test subject used Amazon. com to verify Shirley Temple’s movie playl- ist because she knew that movie reviews often include the list of all an actor’s movies. Another test subject used Google to locate Web sites for primary content or to verify or formulate the structure of a question, as if to define the universe of what was out there in that particular topic. She searched Google using the keywords “ecoterrorism” and “punk rockers.” During exploration of some of her Google results, she learned the names of individuals involved in both movements. She proceeded to search those names via Google to locate biographical information with some success and articulated that the possibly dubious sources behind some Web sites still 354C&RL News June 2006 http:Amazon.com supplied information that was good enough to achieve the task at hand. Critique of methodology It is important to note we have some room for improvement in our use of contextual inquiry methodology for this project, mostly due to logistical constraints. Test subjects were not chosen randomly from the entire undergradu- ate student population; one third were work- ers employed by the library. Interviews were conducted in conference rooms, not the native work environment. Interviews were conducted several weeks after the activities had occurred, which might lead to idealization or omission of the students’ process. At the end of the proto- col the test moderator forced a switch over to the University of Rochester’s federated search implementation if the test subject had not gone there spontaneously during the interview, which disrupted the flow of the session and increased the students’ attention to that part of the library system. Lastly, contextual inquiry was designed for the general hardware and software develop- ment community in the computer industry and libraries may need different methods to support teaching, learning, and research. Future directions for federated search Several design considerations emerged from these interviews which would require coop- erative development to incorporate. Federated search, as a “search all” concept, is useful for users who perceive lists of databases as identical resources. Specific databases known by name, learned about via a professor or library instruc- tion, have more value than a generic “search all” resource. Federated search would have the most value in a high- context environment across sources of equal regard by users. Regarding the presentation of results, the system could provide fewer proprietary func- tions that end up duplicating desktop or browser functions but have a different look and feel. Also, our experience with users of federated search indicates more functions or descriptive metadata to aid in evaluating the relevance of items retrieved in a search attempt would be highly appreciated.7 Notes 1. The University of Rochester is a cus- tomer of both Endeavor and ProQuest, and a member of the Association for Research Librar- ies. Approximately 7,900 undergraduate and graduate students are enrolled on the River Campus, where these interviews took place. 2. Hugh Beyer and Karen Holtzblatt, Con­ textual Design: Defi ning Customer­Centered Systems (Morgan Kaufmann Publishers, San Francisco, 1998). This is a good source for learning more about this method. 3. Rochester’s standard practice is to e-mail a group of students who have previously ex- pressed interest in participating in Rochester Libraries’ Web design process. Selection is on a fi rst-come, first-served basis. For a one hour session, $25 is our standard incentive. 4. More information on Morae software, produced by TechSmith, can be found at www.techsmith.com/products/morae /default.asp 5. NoodleBib, the flagship product of NoodleTools, www.noodletools.com is an online citation manager for K–12 and col- lege- level students. NoodleBib allows users to easily create MLA Works Cited or APA References documents. 6. See www.lib.rochester.edu/index. cfm?page=13&Subject=NSC for the exact page that the test subject commented upon. 7. Project members included the following individuals. Brenda Reeb is a member of the University of Rochester Libraries’ Usability Team; Nora Dimmock, member of Rochester’s Usability Team, and Nancy Foster, lead anthro- pologist for the University of Rochester’s work practice studies, advised the group; Michael Visser was product manager for Encompas for Resource Access, Endeavor Information Systems and is currently product manager for dissertations and theses at ProQuest In- formation and Learning; John Law, ProQuest Information and Learning, is responsible for strategic planning and defining new features for various technologies, including federated search; John D’Ignazio is now a doctoral student at Syracuse University’s School of Information Studies. June 2006 355 C&RL News www.lib.rochester.edu/index http:www.noodletools.com www.techsmith.com/products/morae