Microsoft Word - ES_Furlan.doc Evidence Based Library and Information Practice 2006, 1:4 60 Evidence Based Library and Information Practice Evidence Summary Library Users Expect Link Resolvers to Provide Full Text While Librarians Expect Accurate Results A review of: Wakimoto, Jina Choi, David S. Walker, and Katherine S. Dabbour. ʺThe Myths and Realities of SFX in Academic Libraries.ʺ The Journal of Academic Librarianship 32.2 (Mar. 2006): 127‐ 36. Reviewed by: Wendy Furlan Liaison Librarian, University of Queensland Library Brisbane, Queensland, Australia E‐mail: w.furlan@library.uq.edu.au Received: 30 August 2006 Accepted: 15 October 2006 © 2006 Furlan. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Objective – To determine how successful the link resolver, SFX, is in meeting the expectations of library users and librarians. Design – Analysis of an online user survey, library staff focus groups, retrospective analysis of system statistics, and test searches. Setting – Two California State University campus libraries in the United States: Northbridge, with over 31,000 students on campus, and San Marcos, with over 7,300 students on campus. Subjects – A total of 453 online survey responses were submitted from library users, 421 from Northbridge and 32 from San Marcos. Twenty librarians took part in the focus groups conducted with library staff consisting of 14 of the 23 librarians from Northbridge (2 from technical services and 12 from public services), and 6 of the 10 San Marcos librarians (3 from technical services and 3 from public services). No further information was provided on the characteristics of the subjects. Methods – An online survey was offered to users of the two campus libraries for a two‐ week period in May 2004. The survey consisted of 8 questions, 7 fixed response and 1 free text. Survey distribution was enabled via a different mechanism at each campus. The Northbridge library offered the survey to users via a pop‐up window each http://creativecommons.org/licenses/by/2.0 Evidence Based Library and Information Practice 2006, 1:4 61 time the SFX service was clicked on, while the San Marcos library presented the survey as a link from the library’s home page. Survey responses from both campuses were combined and analysed together. Focus groups were conducted with librarians from each campus library on April 20th, 21st, and 29th, 2004. Librarians attended focus groups only with others from their own campus. Statistics were gathered from each campus’ local SFX system for the 3‐month period from September 14, 2004, to December 14, 2004. Statistics from each campus were combined for analysis. The authors also conducted 224 test searches over the 3‐ month period from July to September, 2004. Main results – Analysis of the surveys revealed that 80% of users expected to see a full‐text article online when they clicked on the SFX button; 20% expected to rarely or never see one. Responses also gave an almost equal split when users were asked if the SFX service met their expectations with 49.5% saying it did and 50.5% saying it did not. The free text survey question asking for comments on the user’s overall opinion of SFX received 174 responses, 26% of which were positive, 40% negative, and 34% mixed. The primary theme expressed in 49% of all comments received was disappointment in not gaining full‐text access. Thirty‐three percent of other comments were classed as having a general theme, while the remaining 19% of comments regarding the SFX service were categorised with themes of complexity, technical problems, efficiency, or confusion. Results from the librarian focus groups differed between the two campuses. Northbridge librarians had 50% to 85% confidence in the accuracy of SFX and were generally impressed with the service. San Marcos librarians had 60% to 100% confidence that SFX would work, however it was also discovered that several participants had suspicions regarding the accuracy of the system. The SFX usage statistics obtained covered 188,944 individual uses of SFX at both campuses. Statistics showed that 48% of these uses resulted in the user clicking on an option provided, either linking to full‐text, catalogue look‐up, or inter‐library loan form access. Of total occurrences, 39.7% had a link to full‐text displayed; this link was accessed 65.2% of the time. Forty‐seven percent of SFX uses provided a catalogue link (23.8% of which were accessed) and 37.9% of uses provided an inter‐library loan link (8.4% of which were accessed). The test searches revealed anomalies to take into account when analysing the SFX usage statistics, including that about 15% of SFX requests display multiple full‐text links. Of the test searches conducted, 22.2% of full‐ text availability results ended in either technical or accuracy errors and 8.8% of catalogue look‐ups produced errors. In those cases where errors did not result there were also significant percentages of instances where the library did not have access to the desired resource: 35.3% of searches correctly indicated that no full text was available, and 57.6% correctly linked to the catalogue to show that the periodical was not held locally. While these are correctly generated system results they are still likely to be seen as unfavourable outcomes by users. Conclusion – The results of the study indicated that both library users’ and librarians’ expectations of SFX were slightly higher than their actual experiences. Librarians’ primary concerns related to the need for more accurate results while library users wanted more full text. It was noted that many complaints associated with SFX were likely to actually be problems with systems that SFX links into rather than the software itself. Although imperfect, SFX, and link resolvers in general, are noted to be a vast improvement on the many separate searches required in the past to locate full Evidence Based Library and Information Practice 2006, 1:4 62 text and undoubtedly user expectations and demand for 100% seamless accessibility will grow. Commentary This study can provide limited insight and a learning opportunity to any library, which has or is considering implementing link resolver software and wishes to examine the likely expectations their users and staff will have of such a system. The sample population for the user survey had no defined criteria and responses were only received from library users who chose to access the online survey. Although the number of responses for the Northbridge campus survey (421) was 13 times that of the San Marcos campus (32), if the estimated student populations of each campus are considered as an estimate of total potential responses, response rates can be estimated as 1.4% for Northbridge and 0.4% for San Marcos. Both figures fall well below statistical significance and results could not be said to be representative of the entire population. The fact that the number of responses for San Marcos was so small brings into question whether those results should have been analysed at all. The authors did not report any statistical significance tests on response numbers prior to analysis of the results, which would have certainly revealed how representative the study was. Furthermore, as the surveys were made accessible via different avenues from each campus library, data collection differences between the two campuses would have had a major impact on not only the number of responses but also on the attributes of the responding population. Given these differences it would probably have been wiser to analyse campus figures independently of one another. The user survey was attached as an appendix for readers to review the content, but the two questions included to gain information about the population were not discussed at all by the authors. This is unfortunate as trends of the different user groups may have provided an interesting insight. Responses to the open‐ended question in the survey were categorised by two of the authors by their positive or negative tone. This is a valid qualitative analysis process, but the calculation of percentage figures from this is highly unscientific and therefore all analysis based on these figures is flawed. When concluding, the authors do state that generalisations can not be made from the results and the “non‐randomized” nature of the survey distribution is acknowledged. The focus groups conducted with librarians at each campus incorporated a good population sample with 61% of Northbridge and 60% of San Marcos librarians taking part. Documentation of further characteristics of the focus group subjects, for example number of years in the profession, could also have enhanced analysis and showed trends in the results. While comments documented from these focus groups are interesting, the absence of the list of questions asked of the groups makes it difficult for readers to put comments into context or to replicate the process. The authors again categorised comments and calculated percentages on this basis, and repeated the unscientific practice from the user surveys. The system statistics discussed provide an insight into the use of SFX, although as with the results from the test searches, do not directly help to meet the study objective of how successful SFX is in meeting the expectations of library users and librarians. Although it poses some possibilities as to why the system may not produce a favourable outcome, it is impossible for system‐generated statistics and author‐ conducted searches to predict or replicate the actual experiences and expectations of users and librarians. The validity of the test Evidence Based Library and Information Practice 2006, 1:4 63 searches is also questionable as searches conducted by information professionals are not necessarily comparable with those performed by the average library user. The lack of detail regarding the searches again makes results difficult for readers to put into context and impossible to replicate. To obtain more reliable and representative results, future studies should consider several factors: maintaining consistency in survey delivery methods; achieving significant response rates from users; gathering, collating and analysing demographical information of subjects (i.e. student or staff level and status, field of study, number of years at the institution); utilising test searches and system statistics retrieved prior to user studies in the formation of relevant and effective survey and focus group questions; not deriving statistics from qualitative data; and ensuring all support documentation are included in final publications. Overall, the study provides an interesting insight into the expectations library users and librarians have for link resolvers and the disparity of these expectations, as library users expect full‐text accessibility and librarians expect an accurate system. However, due to the limitations of the methodology and analysis, further evidence is warranted if decision‐making regarding the implementation of link resolver software and how to best meet or investigate user expectations is to be based on such research.