Evidence Based Library and Information Practice Evidence Based Library and Information Practice 2011, 6.2 36 Evidence Based Library and Information Practice Evidence Summary Users of Virtual Reference Are More Satisfied with the Service They Receive than the Providers of that Service Think They Are A Review of: Hansen, D., Johnson, M., Norton, E., & McDonough, A. (2009). Virtual provider pessimism: Analysing instant messaging reference encounters with the pair perception comparison method. Information Research, 14(4). Retrieved from http://informationr.net/ir/14-4/ paper416.html Reviewed by: Carol D. Howe Reference Librarian/Assistant Professor Gabriele Library Immaculata University Immaculata, Pennsylvania, United States of America Email: chowe@immaculata.edu Received: 23 Feb. 2011 Accepted: 29 Mar. 2011 2011 Howe. This is an Open Access article distributed under the terms of the Creative Commons-Attribution- Noncommercial-Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. Abstract Objective – To examine provider pessimism by comparing user and provider perception of the same instant messaging reference transaction. Design – Instant messaging users and providers completed post-reference transaction surveys which were analyzed using the pair perception comparison method. Setting – A large research university in the United States. Subjects – Two hundred undergraduate journalism students (users of the instant messaging service) and 51 Master of Library Science (MLS) students enrolled in a reference services class (providers of the instant messaging service). Methods – The authors created a research help webpage from which users could access the instant messaging service. Prior to service availability, providers received reference instruction and demonstrated reference aptitude through in-class activities. The authors briefed providers on the project and provided a wiki containing resources they might need during reference transactions. http://informationr.net/ir/14-4/%20paper416.html� mailto:chowe@immaculata.edu� Evidence Based Library and Information Practice 2011, 6.2 37 Providers worked in two-hour shifts, and two providers were available during each shift. The service was available for one week while potential users completed a journalism assignment. The authors asked both users and providers of the service to complete an online survey at the conclusion of the reference transaction. Users and providers completed different surveys, but both types included the following four elements: questions to aid in matching a user to a provider; questions about satisfaction with the service based on guidelines put forth by the Reference and User Services Association (RUSA); open-ended questions about the reference transaction; and questions regarding demographics, prior reference service usage, and knowledge of instant messaging. There were 55 valid reference transactions, and from those, the authors matched 26 pairs of user and provider surveys. The authors analyzed paired surveys to (a) compare the user’s perception of the reference transaction with the provider’s guess about the user’s perception and (b) compare the provider’s self-perception of the reference transaction with the provider’s guess about the user’s perception. The authors introduced the pair perception comparison method for the analysis using two-tailed paired t-tests and Wilcoxon signed-rank tests. Main Results – Analysis of background information showed that users were younger on average than providers and used instant messaging more frequently. Even so, most users and providers felt comfortable with instant messaging. When providers were asked to guess how satisfied overall they thought the user was with the reference transaction, they reported on average that the user was less satisfied than the user actually was. These results were statistically significant. The authors found no significant difference between the providers’ overall satisfaction with the service they provided and their guesses about the users’ overall satisfaction. The authors also analyzed the matched pairs on 14 specific aspects of satisfaction gathered from surveys. When comparing the users’ satisfaction with the service they received and the providers’ guesses about the users’ satisfaction, the providers underestimated the users’ satisfaction on average for all 14 dimensions. The authors found statistically significant differences with regard to 7 of the 14 dimensions: tempo, ease of use, friendliness, understanding, accuracy, follow up, and spelling. When comparing the providers’ satisfaction with a given reference transaction and their guesses about the users’ satisfaction with the same reference transaction, the authors found significant differences for 3 of the 14 dimensions: interest, resource type, and accuracy. Conclusion – This study has shown for instant messaging reference what other studies have shown for face-to-face reference—that provider pessimism exists. Whatever the environment, providers of reference tend to judge themselves more harshly than the people they are helping judge them. Based on a review of the literature, the authors further note that both expert and novice reference providers experience such pessimism. The authors are hopeful that providers will view these results as evidence of their own competence during instant messaging reference transactions. The results of this study provide valuable information for training instant messaging providers. For example, the fact that providers thought users were less satisfied with the tempo of the reference transaction than they really were suggested to the authors that instant messaging providers need not be so concerned about giving a quick answer. An accurate answer is more desirable. In the same vein, providers thought that users were more concerned with spelling than they really were. Both of these cases, and others gleaned from the results, provide insight into what aspects of the reference transaction providers should spend their time and effort on. Finally, the authors introduced the pair perception comparison method to compare feedback from matched pairs on individual Evidence Based Library and Information Practice 2011, 6.2 38 reference transactions, a methodology not used in any earlier studies. They deemed this method to be an effective way to uncover biases and false assumptions. Commentary The results of this study will be of interest to librarians who provide instant messaging reference service, or train others to do so, for several reasons. First, they indicate what users value in a virtual reference transaction. Second, the study results may serve to bolster providers’ confidence. Third, the authors provide an insightful discussion of the pair perception comparison method. Much of the value of this study lies in its unique approach. While other studies have compared user and provider feedback for reference transactions, most of them have looked at face-to-face reference, not instant messaging, and have not matched user and provider responses for a given transaction. Also novel was the assessment of provider satisfaction on two levels: satisfaction with their own performance as well as their guesses about user satisfaction. The study also has limitations that must be addressed. First is the size of the sample; the authors only examined 26 pairs of subjects. This was large enough to test for statistical significance between subjects, but likely too small to provide reliable results in other respects. For instance, are all users of instant messaging cavalier about spelling, or just the small number of users in this sample? Additional studies are needed to validate their results. Another limitation comes from the authors’ hand-picking their subjects. The providers were MLS students in a reference services course. Could this particular class of students generally lack self-confidence? With regard to the users, is it possible that some personality characteristic of journalism students in general skewed the results? It would be interesting to see the results from randomly selected samples in future studies of this sort. The authors introduced another limitation by virtue of using inexperienced providers. They note that provider pessimism has been observed for both new and expert reference providers, but it is not clear whether inexperienced reference providers exhibit more (or less) provider pessimism than their expert colleagues. Additional research to compare new and expert providers is needed. The authors acknowledge that theirs is only one, small-scale study and welcome others to use the pair perception comparison method on different populations for further research. As a whole, this study is a valuable contribution to the existing literature on virtual reference service. The results will hearten any librarian who has experienced self-doubt during an instant messaging reference transaction.