Microsoft Word - ES_1774 Evidence Based Library and Information Practice 2008, 3:3 61 Evidence Based Library and Information Practice Evidence Summary Undergraduates Prefer Federated Searching to Searching Databases Individually A Review of: Belliston, C. Jeffrey, Jared L. Howland, & Brian C. Roberts. “Undergraduate Use of Federated Searching: A Survey of Preferences and Perceptions of Value-Added Functionality.” College & Research Libraries 68.6 (Nov. 2007): 472-86. Reviewed by: Genevieve Gore Coordinator, eScholarship@McGill McGill University Library Montreal, QC, Canada E-mail: genevieve.gore@mcgill.ca Received: 28 May 2008 Accepted: 13 July 2008 © 2008 Gore. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Objective – To determine whether use of federated searching by undergraduates saves time, meets their information needs, is preferred over searching databases individually, and provides results of higher quality. Design – Crossover study. Setting – Three American universities, all members of the Consortium of Church Libraries & Archives (CCLA): BYU (Brigham Young University, a large research university); BYUH (Brigham Young University – Hawaii, a small baccalaureate college); and BYUI (Brigham Young University – Idaho, a large baccalaureate college) Subjects – Ninety-five participants recruited via e-mail invitations sent to a random sample of currently enrolled undergraduates at BYU, BYUH, and BYUI. Methods – Participants were given written directions to complete a literature search for journal articles on two biology-related topics using two search methods: 1. federated searching with WebFeat® (implemented in the same way for this study at the three universities) and 2. a hyperlinked list of databases to search individually. Both Evidence Based Library and Information Practice 2008, 3:3 62 methods used the same set of seven databases. Each topic was assigned in random order to one of the two search methods, also assigned in random order, for a total of two searches per participant. The time to complete the searches was recorded. Students compiled their list of citations, which were later normalized and graded. To analyze the quality of the citations, one quantitative rubric was created by librarians and one qualitative rubric was approved by a faculty member at BYU. The librarian- created rubric included the journal impact factor (from ISI’s Journal Citation Reports®), the proportion of citations from peer- reviewed journals (determined from Ulrichsweb.com™) to total citations, and the timeliness of the articles. The faculty- approved rubric included three criteria: relevance to the topic, quality of the individual citations (good quality: primary research results, peer-reviewed sources), and number of citations. Data were then analysed using ANOVA and MANOVA. Finally, librarians at the ACRL 13th National Conference Presentation were polled about their perceptions of the time savings of federated searching, whether the method meets undergraduates’ information needs, undergraduate preference for searching, and the quality of citations found. Main Results – Seventy percent of all participants preferred federated searching. For all schools combined, there was no statistically significant difference between the average time taken using federated searching (20.34 minutes) vs. non-federated searching (22.72 minutes). For all schools combined, there was a statistically significant difference in satisfaction of results favouring federated searching (5.59/7 vs. 4.80/7 for non-federated searching, α = .05). According to the librarian-created rubric, citations retrieved from federated searching were a statistically significant 6% lower in quality than citations retrieved from non-federated searching (α = .05). The faculty-approved rubric did not detect a difference in the quality of the citations retrieved using the 2 methods. Librarians’ perceptions as assessed at the ACRL 13th National Conference Presentation generally matched the authors’ findings. Conclusion – Overall, students in this study preferred federated searching, were more satisfied with the results of federated searching, and saved time (although the savings were not statistically significant). The quality of citations retrieved via both methods was judged to be similar. The study provides useful information for librarians interested in users’ experiences and perceptions of federated searching, and indicates future studies worth conducting. Commentary This article includes a good review of the current state of federated searching as well as the literature related to it. Overall, the study provides useful results in an area about which many assumptions have been made in the literature but few user surveys have been soundly conducted to validate them. The study used a strong research methodology to analyze undergraduates’ experiences and perceptions of federated vs. non-federated searching. The population was limited to undergraduates and the results may not therefore be generalizable, although the authors speculate that the results would hold. The context of the study (i.e. its implications for the renewal or non- renewal of the WebFeat® licence within the CCLA) may have introduced an element of bias. Participants were randomly contacted but it is unclear how randomization was done or whether there was a self-selection bias in the type of respondent who agreed to participate (response rates were not provided). The number of participants in Evidence Based Library and Information Practice 2008, 3:3 63 each group was sufficient to conduct statistical tests of significance on the data. Written instructions were given to participants to minimize inter-observer and intra-observer bias (it is unclear how many observers were used, however). Instruments were included in appendices, although no screenshot of the federated search implementation was provided and the authors acknowledged that the implementation could have had an effect on the results of the study. The survey questions were clearly posed. IT infrastructure differences between the institutions was not addressed as a possible source of noise when comparing time savings between the two methods. It is unclear whether the two rubrics used to analyze quality were validated, and inter- rater reliability was not assessed for the faculty-approved rubric: it fell upon one undergraduate biology student to assess the quality of the citations retrieved. One could ask why the librarian-created rubric used the impact factor for journal citations instead of using citation counts for each article, although citation counts would have required normalization for publication date. In addition, one might ask why the statistically significant difference favouring individual database searches over meta- searching, captured by the librarian-created rubric, was essentially dismissed by the authors in the discussion section. It is also unclear whether the number of databases being searched affected the participants’ perceptions of federated vs. non-federated searching. The number was based on the average number of databases listed by subject librarians and assumed to be optimal. With respect to the librarian survey conducted at the ACRL conference, it is unclear how the respondents were chosen from the audience. This research contributes to our understanding of undergraduates’ experiences and perceptions of federated vs. non-federated searching. This study is much needed given the growing presence of federated search tools in libraries and can provide empirical support to institutions thinking about purchasing such a system for their own library.