Evidence Based Library and Information Practice Evidence Based Library and Information Practice 2010, 5.4 125 Evidence Based Library and Information Practice Classic Accurate Answers to Reference Queries May Be Provided Less Frequently Than Expected A Review of: Hernon, P., & McClure, C. (1986). Unobtrusive reference testing: The 55 percent rule. Library Journal, 111(7), 37-41. Reviewed by: Eamon C. Tewell Senior Library Assistant Moore College of Art & Design Philadelphia, Pennsylvania, United States of America Email: etewell@moore.edu Received: 21 June 2010 Accepted: 30 Sept. 2010 2010 Tewell. This is an Open Access article distributed under the terms of the Creative Commons-Attribution- Noncommercial-Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. Abstract Objective – To determine the number of government documents reference questions that are answered correctly by professional library staff. Design – The authors utilized unobtrusive reference testing: reference queries posed to library personnel who were unaware they were being evaluated. As opposed to other designs that require the researcher’s presence in the setting, unobtrusive testing utilizes proxies to administer test questions to the subjects, reducing the possibility of reporter bias. Setting – Twenty-six public and academic libraries participating in the U.S. Government Printing Office Depository Program located in the Western, Southern, and Midwestern United States. The Federal Depository Program consisted of 1400 libraries at the time of the study. One public and one academic library were chosen for each city. Subjects – Reference and government documents librarians. These two staff types were selected in order to compare the accuracy of each group’s responses to the queries. Methods – A set of 15 predetermined factual and bibliographic questions were developed by the authors and administered to library staff respondents by proxies. Government documents were selected as the foundation for the test questions. In selecting federal depository libraries for their sample the authors could ensure all queries may hypothetically be answered using U.S. Government Printing Office documents, as all mailto:etewell@moore.edu� Evidence Based Library and Information Practice 2010, 5.4 126 of the libraries would hold the resources in question. Graduate students enrolled in the University of Arizona and University of Oklahoma library science programs were trained by the authors to serve as proxies. The proxies posed as library users and administered the set of queries at each selected library. Reference librarians and government documents librarians were tested separately, receiving seven and eight questions respectively at each library visited. Over a four-month period a total of 390 questions were posed and their answers recorded. Main Results – The respondents correctly answered 241 of 390 queries (62 percent). Government documents librarians accurately answered 65 percent of questions, while reference librarians successfully responded to 59 percent. Hernon and McClure derived the “55 percent rule” for reference accuracy from these results and previous unobtrusive studies conducted by both the authors and other researchers. This body of research estimates the rate of accurate answers of factual and bibliographic questions to be between 50 and 62 percent. Data regarding the “interview and search process” (I&S), defined as the activities between the time a query was posed and when a resolution was provided, also yielded intriguing findings. Regardless of the question asked, the average I&S duration was three to five minutes. Two-thirds of the accurate answers were supplied within three minutes, and 89 percent within five minutes of the initiation of I&S. The duration of I&S did not vary significantly by library type or librarian type. Reasons for the provision of inaccurate answers included providing the wrong data (64 percent of instances), responding with “don’t know” and ending the interaction (20 percent), or claiming the library did not own a source that would answer the query (15 percent). Other findings included the fact that respondents infrequently offered referrals, which took place in 17 percent of all interactions, and that the three geographic regions studied had an even distribution of correct answers. Conclusions - Based on their research results as well as those of similar unobtrusive studies, the authors propose two rules regarding reference assistance that apply to public and academic libraries. First, reference librarians correctly answer approximately 55 percent of factual and bibliographic queries received (the 55 percent reference rule). Second, librarians spend no more than five minutes on most factual and bibliographic questions (the five- minute answer rule). Ultimately, Hernon and McClure recommend reevaluating the centrality of reference services to library operations. If trained professionals are unable to answer more than 60 percent of factual questions correctly, should reference services continue to receive such considerable staffing and funding? Or, alternatively, should libraries increase efforts to improve the accuracy of answers? The authors close by challenging the profession to address the need for strategic assessment of reference effectiveness. Commentary The accuracy and delivery of reference services remains a focal point for public and academic libraries, and as an area involving substantial resources and staff time it demands informed decision-making. The authors’ goal was to emphasize the user’s perspective as a means to examine widely held assumptions about reference services. Hernon and McClure’s ultimate aim, however, appears to have been to encourage practicing librarians to apply empirical evidence to critical issues in the profession (1987, p. 283). In many ways, their work is an improvement on the authors’ past studies utilizing unobtrusive testing methodology. Their approach is commendable, yet the study Evidence Based Library and Information Practice 2010, 5.4 127 contains several issues of note and potential limitations: • The graduate student proxies were instructed to pose questions to the individual they presumed to be the reference librarian, resulting in the possibility of incorrect respondents (such as support staff) and erroneous data. A more rigorous design would include a second observer to corroborate proxy observations and minimize observer biases. • Accuracy is presented as the sole indicator of quality reference service. In practice, a number of other factors not addressed by accuracy contribute to excellent service from the user’s perspective. • Not all varieties of reference questions are represented by the factual and bibliographic queries posed. Queries requiring ambiguous answers, for example, are not easily measured and might be left unaddressed. • A potential bias was introduced because the government documents sought were not identified as such, yet government document librarians’ expertise could result in identifying these requests. • Government documents inquiries, the foundation for the authors’ test questions, are more likely to require additional expertise to successfully navigate compared to standard reference questions. However, it should be noted that all of the resources requested were indeed accessible through the respective library’s catalogue, and could potentially have been located by the librarians. • Brief, non-routine factual questions such as those developed by the authors do not constitute the majority of questions received at reference desks, possibly resulting in a lower accuracy rate compared to that which might be found while observing everyday reference operations. • Schumacher correctly notes that the inclusion of routine factual and directional questions would cause the accuracy rate to increase dramatically (1987). While their research contains these aforementioned flaws, Hernon and McClure succeeded in creating sustained discussion on the topic of improving the evaluation of reference services. Their landmark study contributed significantly to the popularity of unobtrusive testing to assess services. Applications of the methodology have been limited to investigating the quality of reference, yet unobtrusive measures could in fact be applied to multiple areas, particularly instruction and outreach. Unobtrusive testing, also frequently referred to as unobtrusive observation, provides unique advantages not afforded by other methodologies. Since research subjects are unaware their responses are being recorded, observation does not change their behaviour. As Whitlach states, subjects “cannot introduce bias into the study because they are unaware of it” (1989). Additional strengths beyond the elimination of self-reporting bias are the significant scope of data that can be collected, such as non-verbal behaviours, and the lack of reliance on the respondent’s recollection of events. Weaknesses of unobtrusive testing include generalizability and ethical concerns. Due to the method’s basis in observation, a sample must be examined numerous times to produce generalizable findings. Generalizability poses a considerable obstacle for unobtrusive studies examining specific populations. A thornier issue is that of ethics. The lack of informed consent, participant unawareness of the study, and potential invasion of privacy can raise significant concerns depending on the research being conducted. On a practical level, the ethical questions raised by unobtrusive observation can make it more difficult for researchers to receive institutional or ethics review board approvals. Evidence Based Library and Information Practice 2010, 5.4 128 As might be expected of a work that is highly critical of library performance and contains major implications for practitioners, “the library community responded to these unobtrusive studies first with shock, then with denial” (Hults, 1992, p. 143). Bailey, a vocal detractor of Hernon and McClure’s methodology, stated that the 55 percent rule “has caused a stir among librarians in this country and has reverberated internationally” (1987, p. 280). In 2005, Hubbertz sharply criticized the article’s methods and findings, arguing that unobtrusive observation is better suited to comparing and ranking libraries than assessing the quality of a service (p. 6), and that the validity of the results was seriously compromised due to test subjects being asked different questions. Numerous articles have responded directly to the findings, testing the validity of the 55 percent rule or altering the methods, with varying results (Durrance, 1989; Whitlatch, 1989; Whittaker, 1990; Jardine, 1995; Richardson Jr., 2002; Hubbertz, 2007). The influence of Hernon and McClure’s study is evidenced in part by future research employing the same methods and applying them to other indicators of patron satisfaction. Durrance, for example, sought to address the larger reference context and measured the “willingness of the inquirer to return to the same staff member at a later time” (1989, p. 32), while Dilevko and Dolan applied the design of Hernon and McClure’s study to a different geographic location (2000). Unobtrusive testing has also surfaced in “mystery shopping” evaluation programs. Originating in the business world and designed to assess employee behaviour, this practice has been adopted by a number of libraries and utilizes the unobtrusive method (Thomas, 2000). Without Hernon and McClure’s controversial findings based on a user-centered research design, it is questionable whether researchers would have worked to develop other measurements of reference effectiveness. Other authors have utilized the framework laid by Hernon and McClure’s study in innovative ways. Curry, attempting to gauge the quality of reference services to a particular marginalized youth population, asked teenagers to pose gay- and lesbian-related questions at twenty public libraries in British Columbia (2005). Shachaf and Horowitz’s 2006 article examined whether the inferred ethnicity of a fictitious patron’s e-mail request affects the response of reference librarians in twenty-three Association of Research Libraries member libraries. While the quality of reference services is still the subject of evaluation, these recent studies demonstrate the potential for additional work in the unobtrusive testing domain. Hernon and McClure’s study does not hold to current evidence based library and information practice standards. As published, the study contains little information allowing for replication. Additional factors, such as potential observer bias and different questions asked of test subjects, diminish the study’s reliability. However, in a 1987 response to the intense debate their article had provoked, the authors’ laudable objective can be seen in the statement: “research may serve as an objective counterweight to ‘informed opinion’—a widespread commodity in librarianship.” Indeed, the greatest contribution of Hernon and McClure’s work is the reminder that assumptions regarding library service merit rigorous study and debate. References Bailey, W. G. (1987). The “55 percent rule” revisited. Journal of Academic Librarianship, 13(5), 280–82. Childers, T. (1987). The quality of reference: Still moot after 20 years. Journal of Academic Librarianship, 13(2), 73-74. Crowley, T., & Childers, T. (1971). Information service in public libraries: Two studies. Metuchen, NJ: Scarecrow Press. Curry, A. (2005). If I ask, will they answer? Reference & User Services Quarterly, 45(1), 65-75. Evidence Based Library and Information Practice 2010, 5.4 129 Dilevko, J., & Dolan, E. (2000). Government documents reference service in Canada: A nationwide unobtrusive study of public and academic depository libraries. Library & Information Science Research, 22(2), 185-222. Durrance, J. C. (1989). Reference success: Does the 55 percent rule tell the whole story? Library Journal, 114(7), 31-36. Hernon, P., & McClure, C. R. (1987). Where do we go from here? A final response. Journal of Academic Librarianship, 13(5), 282-284. Hubbertz, A. (2007). The fallacy in the 55 percent rule. DttP: A Quarterly Journal of Government Information Practice & Perspective, 35(3), 15-17. Hubbertz, A. (2005). The design and interpretation of unobtrusive evaluations. Reference & User Services Quarterly, 44(4), 327-335. Hults, P. (1992). Reference evaluation: An overview. The Reference Librarian, 17(38), 141-150. Jardine, C. W. (1995). Maybe the 55 percent rule doesn’t tell the whole story: A user- satisfaction survey. College & Research Libraries, 56(6), 477-85. Richardson, J. V., Jr. (2002). Reference is better than we thought. Library Journal, 127(7), 41-42. Schumacher, M. (1987). The continuing debate on library reference service: A mini- symposium. Journal of Academic Librarianship, 13(5), 278-79. Shachaf, P., & Horowitz, S. (2006). Are virtual reference services color blind? Library & Information Science Research, 28(4), 501- 520. Thomas, J. (2000). Mystery shoppers at the library: A planning report. California State University Long Beach Library. Retrieved 20 May 2010 from: http://www.csulb.edu/divisions/aa/grad _undergrad/senate/committees/assessm ent/dev/awards/documents/thomas_99. pdf Whitlatch, J. B. (1989). Unobtrusive studies and the quality of academic library reference services. College and Research Libraries, 50(2), 181-194. Whittaker, K. (1990). Unobtrusive testing of reference enquiry work. Library Review, 39(6), 50-54. / Evidence Based Library and Information Practice