Evidence Based Library and Information Practice 2009, 4:2 152 Evidence Based Library and Information Practice Evidence Summary Libraries Demonstrate Low Adherence to Virtual Reference Service Guidelines A Review of: Shachaf, Pnina, and Sarah M. Horowitz. “Virtual Reference Service Evaluation: Adherence to RUSA Behavioral Guidelines and IFLA Digital Reference Guidelines.” Library & Information Science Research 30.2 (2008): 122-37. Reviewed by: Elise Cogo Information Specialist Consultant Ottawa, ON, Canada Email: ecogo@uwo.ca Received: 16 December 2008 Accepted: 02 April 2009 © 2009 Cogo. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Objectives – This study evaluates the level to which virtual (asynchronous e-mail) reference services adhere to professional guidelines. Specifically, it addresses the following research questions: 1) To what extent do virtual reference services adhere to the American Library Association (ALA) Reference and User Services Association (RUSA) and the International Federation of Library Associations (IFLA) guidelines? 2) How does the level of adherence to RUSA or IFLA guidelines vary based on request type, user name, and institution? 3) Is there a correlation between outcome measures of reference transactions (accuracy, completeness, and satisfaction) and the level of adherence to RUSA or IFLA guidelines? Design – Unobtrusive evaluation of researcher- generated queries. Setting – Fifty-four academic libraries in North America. Subjects – A total of 324 queries were sent to the 54 libraries, with each library receiving six different types of requests from six different user names. Methods – Researchers developed two coding schemes for the guidelines (34 codes and 12 attributes for the RUSA guidelines and 33 codes and 10 attributes for the IFLA guidelines). Each of the six user names used represented an ethnic and/or religious group identity: Mary Anderson (Caucasian, Christian), Moshe Cohen (Caucasian, Jewish), Ahmed Ibrahim (Arab), mailto:ecogo@uwo.ca Evidence Based Library and Information Practice 2009, 4:2 153 Latoya Johnson (African American), Rosa Manuz (Hispanic), and Chang Su (Asian). The six request types were designed so that three would be answered (questions 1-3) and three would be out of scope and not answered (questions 4-6). The following queries were sent, individualized for each institution: 1) Dissertation query; 2) Sports team query; 3) Population query; 4) Subject query; 5) Article query; 6) Request for a PDF copy. The 324 queries were uploaded into NVivo 2 software, and all e-mail transactions were coded and analyzed. Main Results – Analysis of the 324 transactions from 54 libraries showed the following results: 1) Low levels of adherence to both sets of guidelines; 2) Varied levels of adherence based on request types and user names on both sets of guidelines; 3) Variation in institutional rank according to different sets of guidelines; 4) No correlation between user satisfaction and adherence to either set of guidelines. Conclusion – This study suggests that higher levels of virtual reference service effectiveness could be achieved by automatically integrating some less observed behaviours (e.g., thank you notes) into replies sent to users and by increasing librarians’ awareness of professional guidelines through training and detailed institutional policies. The authors also suggest that librarians should be aware of their tendencies to react differently to different user groups, and that administrators can facilitate this by providing diversity workshops. Commentary One of the primary strengths of this study is that it provides a detailed methodology for systematically analyzing transactions in light of ideal professional standards, and it provides an empirical benchmark for evaluating virtual reference services and for objective comparison with other libraries. The authors note that a few previous evaluative studies of virtual reference services have used these professional guidelines but only parts of the guidelines were used and those that were used were not treated systematically. The article includes many tables with the coding schemes used and transaction results and frequencies, thereby providing clear and transparent methods and results. The large number of libraries and transactions evaluated in the study adds strength to the key finding that there was an overall low level of adherence to professional guidelines. The study also aimed to evaluate the effects of several reference query variables (e.g., request type and user name,), which were included in the interpretation of the results. The use of NVivo software to facilitate the qualitative analysis allowed for the coding of text as well as the linking and modeling of qualitative data. A detailed coding scheme was developed to represent each guideline criterion in order that systematic interpretations and analyses could be made. This allowed a large amount of qualitative information to be analyzed quantitatively and for key statistics to be generated. In addition, the software’s searching and co-occurrence capabilities allowed for better understanding of the frequencies that were generated from the transactions. One limitation that the authors noted is that some of the frequencies need to be interpreted with caution since they may not apply in each situation. For example, certain request types may violate institutional policy. Other limitations of this study, which the authors suggest as avenues for future research, include the inability to assess real information needs, real-time interactions, or user affiliation. One of the implications of this study is the need for further research and training within the library profession with regard to service to various ethnic and/or religious groups. Overall, this study makes a very significant contribution Evidence Based Library and Information Practice 2009, 4:2 154 to the methodology of virtual reference service evaluation and provides a benchmark for the status of virtual reference service guidelines adherence in academic libraries.