Editorial Evidence Based Library and Information Practice 2009, 4:2 133 Evidence Based Library and Information Practice Article Study in Grey and White: Measuring the Impact of the 8Rs Canadian Library Human Resources Study Allison Sivak Assessment Librarian University of Alberta Libraries University of Alberta Edmonton, Alberta, Canada asivak@ualberta.ca Received: 21 February 2009 Accepted: 8 May 2009 © 2009 Sivak. This is an Open Access article distributed under the terms of the Creative Commons Attribution License, , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Objective – To use the 8Rs Canadian Library Human Resources Study (the 8Rs Study) as a test case to develop a model for assessing research impact in LIS. Methods – Three different methods of citation analysis which take into account the changing environment of scholarly communications. These include a ‚manual‛ method of locating citations to the 8Rs Study through a major LIS database, an enhanced-citation tool Google Scholar, and a general Google search to locate Study references in non-scholarly documents Results – The majority of references (82%) were found using Google or Google Scholar; the remainder were located via LISA. Each method had strengths and limitations. Conclusion - In-depth citation analysis provides a promising method of understanding the reach of published research. This investigation’s findings suggest the need for improvements in LIS citation tools, as well as digital archiving practices to improve the accessibility of references for measuring research impact. The findings also suggest the merit of researchers and mailto:asivak@ualberta.ca Evidence Based Library and Information Practice 2009, 4:2 134 practitioners defining levels of research impact, which will assist researchers in the dissemination of their work. Introduction The question of research impact is one of concern to scholars, universities, and research funding bodies. Researchers and the organizations or associations which support their work face increasing demands to demonstrate the impact of their research as a measure of accountability, particularly when that work is supported through public funds. With respect to professional achievement, the impact factor of the journals in which pre-tenured academics publish has influence in their application for tenure (Monastersky 2005). Further, leaders within the open access movement promote freely-available online scholarly content as a method of ‚seamlessly linking data, knowledge, and scholars‛ which will ‚stimulate and accelerate discovery — and ultimately to fuel advances beyond the realm of scholarship‛ (Association of Research Libraries 2009). The 8Rs Canadian Library Human Resources Study (the 8Rs Study) is a research project that can serve as a test case to develop a model for assessing research impact in Library and Information Studies (LIS). The study has a number of characteristics that make it uniquely suited to this purpose. First, the study was designed to answer emerging concerns from practitioners regarding retirements, recruitment, and other aspects of human resources. Second, the 8Rs Research Team conceived the study as a tool which could inform library administrators’ and managers’ decision-making regarding human resource issues; therefore, the report presented not just major findings, but also presented recommendations for specific stakeholders within the library field, i.e., administrators, practitioners, educators, and associations. Third, the work was financially supported by the library community and both the federal and provincial government bodies, which has implications for accountability, although certainly not identical implications to those of researchers funded by formal funding agencies. Last, study results have been disseminated in the form of grey literature, including reports freely available online via the study’s website at and distributed in print and conference presentations for which slides were sometimes posted. The implications of the study’s characteristics are that its relevance to the professional library community should be high and the results are accessible; both these implications would ostensibly increase the study’s impact. 8Rs study context The 8Rs Canadian Library Human Resources Study has been a major research project in Library and Information Studies. It was conducted by a research team at the University of Alberta Libraries, and was designed at a time of great anecdotal discussion in the library community regarding anticipated ‚mass retirements‛ of senior librarians. The work was inspired in part by the previous research of Stanley Wilder (2003) on the demographics of librarians at member institutions of the Association of Research Libraries (ARL). A notable finding in Wilder’s study was that academic librarians, as a profession, were older than those in other fields, noting that 48% of North American librarians in ARL libraries would be at retirement age by 2015. Wilder’s statistics gave an even more pessimistic picture for the Canadian library community, suggesting that a greater percentage of Canadian academic librarians http://www.ls.ualberta.ca/8rs/ Evidence Based Library and Information Practice 2009, 4:2 135 were older, on average (2003). Where Wilder noted that 23% of American research librarians were under age 40, only 16% of Canadian research librarians were in this age category (2003). An article published around the same time in the book trade publication Quill & Quire, reviewed published Labour Force Statistics from Statistics Canada to project retirement figures for the industry. The article’s title was alarmist in nature: ‚Libraries Face ‘Skills Gap’: 48% of Librarians Could Retire by 2005‛ (Weiler 2000). From 2002 to 2005 the 8Rs research team conducted a study of human resources in Canadian libraries across all sectors (academic, public, and special). After the initial development of the research proposal, the research team concluded that the study required several types of support from the library community: financial support, in- kind support (e.g., membership lists for the sampling frame), and community ‚buy-in‛ to encourage survey response. Research team chair Ernie Ingles made a number of presentations on the proposed work at conferences and association meetings. Data collection began in 2003, and several methods were used: interviews with 17 library administrators, focus groups with administrators of research libraries and large urban public libraries, and most substantially, major surveys of library administrators and library staff (both librarians and paraprofessionals). Response rates overall were approximately 36%, resulting in data from 461 administrators and 4,693 library staff. The 8Rs report, The Future of Human Resources in Canadian Libraries, was published in the spring of 2005, by the University of Alberta Libraries, in both French and English. This report falls into the category of grey literature. Rabina (2008) reviews several definitions of grey literature, noting that ‚there is agreement on the main characteristics of grey literature: they are materials that are published by not for profit institutions, and, as a result, typically are not marketed or distributed by commercial publishing organizations‛ (34). This above definition of grey literature is appropriate with respect to the character of the study’s publisher, the University of Alberta Libraries, as a non-profit organization for which publishing is an atypical activity. However, the 8Rs Study is an unusual instance of grey literature publication because the Report was widely marketed and distributed. It was made freely available in PDF format on the 8Rs website , and some print copies were produced in order to facilitate ease of reading; at 275 pages, the team believed that print would be the preferred format for some readers. The publication was formally launched at the annual conference of the Canadian Library Association in June 2005; print copies of the report were available for purchase, and the 8Rs Research Team made a presentation on the study’s major findings. In July 2005, over 300 print reports were distributed to members of the Canadian Association of Research Libraries, the Canadian Association of Large Urban Public Libraries (now the Canadian Urban Libraries’ Council), the Provincial and Territorial Library Directors Council, the Canadian Masters of Library and Information Studies (MLIS) programs, and to all other organizations and associations that supported the study through financial and in-kind assistance. Additionally, copies were mailed to the membership of the Association of Research Libraries (ARL) and the Association for Library and Information Science Education. This resulted in further promotion regarding the 8Rs Study through third http://www.ls.ualberta.ca/8rs/reports Evidence Based Library and Information Practice 2009, 4:2 136 parties. Shortly after the report’s publication, the Canadian Library Association (CLA), in conjunction with the 8Rs Research Team, announced the formation of a President’s Council on the 8Rs to further investigate implications of the study’s findings and recommendations. One of the President’s Council actions was to promote the study; to this end, the group embarked on several different initiatives to present relevant findings to different stakeholders in the Canadian library community. For example, while many statistics in the report were broken down by library sector and sub-sector, the sheer size of the document was presumed to be daunting. Therefore, the group determined a need for ‚executive summaries‛ of findings for different library sectors such as research libraries, other academic libraries, special libraries, urban public libraries, other public libraries, etc. The report’s publication was also noted in the June 2005 issue of the ARL Bimonthly Report. Over the following months, members of the Research Team presented the study’s major findings to a number of audiences, predominantly at library association conferences. These presentations were tailored towards their audiences, in that rather than presenting the aggregate statistics for the Canadian library community as a whole, the presentation focused on those statistics more closely related to the sector hosting the conference. A measure of the success agreed upon by the 8Rs Study research team would be its ‚use‛ by library staff and scholars in workforce planning, as well as in understanding the current and predicted workforce. The study described below endeavors to measure this success. Aims This paper investigates the impact of the 8Rs Study through three different methods of citation analysis. It attempts to take into account the changing environment of scholarly communications by using a ‚manual‛ method of locating citations to the 8Rs Study through a major LIS database, through the enhanced-citation tool Google Scholar, and through a general Google search to locate 8Rs Study references in non- scholarly documents. Through the analysis, it attempts to develop recommendations for understanding the impact of LIS research in the field. Methods A number of parameters were defined prior to beginning the search: references to the study in any article or presentation given by the 8Rs Research Team were not counted. References by other library scholars who had connection to the 8Rs Study (members of the CLA President’s Council, for example) were counted. The reason for this was that presentations given by the 8Rs Research Team served two pre-defined functions: to both raise awareness of the 8Rs Study, and to disseminate the results for relevant audiences. For the purposes of this study ‚impact‛ refers to the utilization of the 8Rs published statistics, report recommendations, or original data to support their research questions. Citation analysis was selected as the most appropriate method for conducting this investigation. Prior to the development of commercial indexing and abstracting vendors, such as Thomson Scientific, Chemical Abstracts, MathSciNet, PsychINFO, and Google Scholar, citation analysis was a laborious and time- consuming process, involving the textual review of publications in periodicals and books to map the frequency of citation of particular authors or papers. Electronic databases provide a vastly-improved process; however, Neuhaus and Daniel’s review notes the importance of critically assessing these databases for their potential with respect to citation analysis as well as Evidence Based Library and Information Practice 2009, 4:2 137 for their coverage limitations (2008). This study also attempts to identify some relevant sources for the purpose of citation analysis in the field of LIS. With this aim, it was important to begin with the Library and Information Science Abstracts (LISA) database, the most credible source for the scholarly literature. In their content analysis of LIS research databases, Koufogiannakis, Crumley, and Slater concluded that LISA provides the best coverage of the ten most research-intensive peer-reviewed journals in the field of LIS (2004). However, Aina’s work on grey literature in LIS stated that although some coverage of grey literature was present in LISA, it did not serve as a comprehensive source (2001). Currently, there exists no LIS- specific citation-enhanced database, meaning that citation analysis is laborious and time-consuming in the LIS databases, leaving Google Scholar as the sole automated citation-assessment tool. Therefore, both these tools were selected to identify citations of the 8Rs Study. In order to understand the scope of the 8Rs study published in the journal literature, LISA was searched both by keyword ‚8Rs,‛ and by the controlled vocabulary subject headings ‚library staff‛ and ‚leadership,‛ limited to the dates 2005-2009. References were checked in all the resulting articles which were accessible to the researcher to see if the articles cited the 8Rs Study. Both peer-reviewed and non-refereed citations were counted. The database searches were complemented by searches in Google and Google Scholar for the combined terms ‚8Rs,‛ ‚library,‛ and ‚human resources.‛ The addition of the search in Google sought to identify any references to the study in the grey literature. Results The search located 480 citations in the LISA database, 3140 in Google, and 161 in Google Scholar. Of the LISA citations, 118 articles could not be obtained, representing 28 individual journal titles. The summary of relevant search findings is shown in Table 1. In all, 68 relevant references were located, with about one- third (22) occurring in articles published in peer-reviewed journals, 19 found in conference presentations online, 6 in non- peer-reviewed journals, 6 in blogs, and 5 in reports. Four references to the publication were found on websites in bibliographies or as ‚suggested resources‛, and four were found related to libraries’ internal planning documents such as strategic plans. Two references to the study were located on MLIS course syllabi. The majority of references (82%) were found using the Google or Google Scholar search engines. The remaining 18% of the references were located using the LISA database. It should be noted that each article was counted only once; the LISA database was the first method of searching employed, and articles located using LISA were not counted in the results for Google or Google Scholar, if found again. Each source had different limitations. Although Google resulted in the greatest number of results, those results needed to be assessed individually for categorization purposes. However, the Google and Google Scholar searches took much less time to conduct than the searches in LISA, as the LISA database does not have an enhanced-citation tool. Two categories of ‚formal‛ publications were not found in LISA: conference presentations and reports. Indeed, the original 8Rs report, The Future of Human Resources in Canadian Libraries, is not indexed in the LISA database. While conference presentations would not normally be expected to be found in a Evidence Based Library and Information Practice 2009, 4:2 138 database, reports are at times indexed in commercial databases. One type of ‚informal‛ publication that can be interpreted as a strong indicator of research impact is the internal planning document for an individual library or library association. Four such publications were found via Google: two strategic planning documents, minutes from a Board of Directors meeting, and one memo to support an institutional human resources initiative. As an original goal for the 8Rs Study was to present findings in a way that allowed institutions to make use of the statistics in their decision-making process, this finding suggests some success. Table 2 analyzes the search findings by year and type of publication. The highest number of references to or citations of the 8Rs Study (29) took place in 2007, which is also when the greater number of articles published (13) in peer-reviewed journals appeared. This was an increase from the number of references found for the years 2005 and 2006 (6 and 18 respectively). In 2008 the number dropped to 15. The peer-reviewed articles tended to be published 18-24 months after the report was released; which seems intuitive, suggesting that some period of time is necessary for the report to be disseminated and further analyzed before other scholars are likely to cite it. Location/Type Conference Non peer- reviewed journal Peer- reviewed journal Report Blog Course syllabus Resource on website Internal planning document Total Google 13 1 6 3 6 2 4 4 39 Google Scholar 6 9 2 17 LISA 5 7 12 Total 19 6 22 5 6 2 4 4 68 Table 1: Number of references to 8Rs Study, by search method and publication type Type 2005 2006 2007 2008 Total Blog 2 4 6 Conference 7 10 2 19 Non peer-reviewed journal 2 1 3 6 Peer-reviewed journal 1 1 13 7 22 Report 1 2 1 1 5 Course syllabus 2 2 Resource on website 1 3 4 Internal planning document 3 1 4 Grand Total 6 18 29 15 68 Table 2: Frequency of reference by type of publication and year published Evidence Based Library and Information Practice 2009, 4:2 139 While this is not a study of the comprehensiveness of the citation sources, it is notable that the search results did not include some conference presentations which cited the 8Rs Study, of which the author was aware. Several explanations are possible here, the most likely being that the presentations may not have been uploaded post-conference. Conference presentations are increasingly being hosted on conference websites, but this practice is clearly not a standard within the field. Table 3 summarizes the type of reference made to the 8Rs Study. The Study’s published statistics were the primary reference point, with 52 of 68 publications citing some of the study’s key findings. Twelve of the publications made reference to the study’s publication in general; this type of reference tended to remark on implications of an aging workforce or comment on major study findings without specific reference to statistics. Three references specifically discussed an aspect of the work underway by the CLA President’s Council on the 8Rs, and in two publications the authors completed their own data analysis, having requested data files from the 8Rs research team. Type of Reference Total Original data analysis 2 Reference to President's Council work 3 Reference to published statistics 51 Reference to study publication 12 Total 68 Table 3: Type of reference made to 8Rs study Type Original data analysis Reference to President's Council work Reference to published statistics Reference to study publication Total Blog 3 3 6 Conference 18 1 19 Course syllabus 2 2 Internal planning document 2 2 4 Non peer-reviewed journal 6 6 Peer-reviewed journal 2 20 22 Report 4 1 5 Resource on website 1 3 4 Total 2 3 51 12 68 Table 4: Frequency of type of reference by type of publication Evidence Based Library and Information Practice 2009, 4:2 140 The frequency of type of reference cross- tabulated with the type of publication found shows that references to the study’s publication, and more specifically to the published statistics, were generally distributed across several types of publications. Original data analyses were only found in peer-reviewed articles, and references to further work on the study were only found on blogs Discussion The methods employed in this study had some limitations. The search resulted in English-language results only, and the keyword search was not replicated using ‚bibliothèque‛ in place of ‚library.‛ Although the majority of library literature is in the English language, a review of the study references in French might add useful information, as this was a Canadian study. In addition, the data was not analyzed for overlap among the LISA database, Google, and Google Scholar, since this study did not intend to formally compare the coverage between different citation sources. However, future analysis of duplication between these sources, in particular between Google Scholar and LISA, would be useful in determining which tool is a more effective resource for citation analysis. The databases ISI/Thomson Web of Knowledge and Scopus were not included as sources for this study, although both have been frequently used in other disciplines for citation analysis. These two databases were excluded from the study due to the limited coverage of LIS journals; while LISA indexes 414 journal titles, Web of Knowledge indexes 62 titles and Scopus indexes 79 titles. A large proportion of scholarly communication within the field of LIS takes place in the form of conference presentations which may not be subsequently published in peer-reviewed journals. Although presentations are certainly less-formal publications as compared to peer-reviewed literature, it can be argued that they are important aspects of scholarly communication; presentations to peers create a forum in which scholars can receive useful responses to their work, which may very well develop or solidify their thinking on the subject. A conference may also serve as an arena in which to test theories or research findings, to determine the feasibility of writing an article for submission to a journal, and may mark an important point in the development of the research. An increasing number of LIS conference presentations are accessible online, via conference websites; most likely in response to demand from delegates, so that they may refer to the work presented in the future. The question remains as to the need for an aggregator of conference content for the field of Library and Information Studies, in order to capture subject matter of concern to the profession. Librarians, other researchers, and associations should consider the consistent archiving of presentation material in digital repositories for this purpose. That the number of references to the 8Rs study decreased in 2008 may allude to the ‚shelf life‛ of a publication or of the perceived timeliness of the statistics. Researchers have established that grey literature citation tends to peak in the first five years after publication, whereas journal literature has a longer ‚citation window‛ of seven to ten years (Di Cesare et al. 2008). This study should therefore be repeated at least six years after the 8Rs Study release. While small in number, the publications that employed original analysis or ‚data repurposing of the 8Rs data represent an important and novel approach to use of the study. Data repurposing is an emerging issue in contemporary scholarship across many disciplines. Ponti (2008), in her proposal of a virtual research Evidence Based Library and Information Practice 2009, 4:2 141 ‚collaboratory‛ for LIS, suggests that ‚collaboratories have the potential to expand the participation of practitioners in research contexts,‛ and proposes data repurposing as a method that could close the education-practice gap identified by the library literature (271). The issues and questions regarding data repurposing are complex and will not be addressed in this paper. However, given that scholars in other disciplines are raising questions about the benefits, concerns, and logistics of data repurposing (Silva 2007, Smith 2008, Sales et al. 2006, Morrison et al. 2009), LIS scholars should investigate the implications of this practice within the field. Researchers have found that the ability of Google Scholar to retrieve different versions of grey literature allows for the reconstruction of the document life cycle; this can provide insight into the path of research publication between grey literature and journal literature (Di Cesare et al. 2008). Further, the Google searches also resulted in some references that allowed for an understanding of the way the 8Rs Study is informing literature outside the peer- reviewed journals, as well as how it is informing practice as noted in organizational documentation. Impact on disciplinary knowledge and future work are general goals of all scholarly research; however, LIS research is integral to the professional work of librarianship, and so impact cannot strictly be measured through the peer-reviewed literature—particularly considering the nature of the study and the research team’s goals in completing this work. Conclusion The research-practice gap is of concern to scholars and practitioners in the field of Library and Information Studies. As many practitioners increasingly aim to base their professional work on evidence-based decision-making, many researchers strive to connect their research to the professional sphere. In-depth citation analysis provides a promising method of understanding the reach of published research. This investigation’s findings suggest the need for improvements in LIS citation tools, as well as digital archiving practices to improve the accessibility of references for measuring research impact. The findings also suggest the merit of researchers and practitioners defining levels of research impact, which will assist researchers in the dissemination of their work. Works Cited 8Rs Research Team. The Future of Human Resources in Canadian Libraries. Edmonton: Univ. of Alberta, 2005. Aina, L. O. ‚Grey Literature and Library and Information Studies: A Global Perspective.‛ International Journal on Grey Literature 1.4 (2000): 179- 182. Di Cesare, Rosa, and Daniela Luzi, Roberta Ruggieri. ‚The Impact of Grey Literature in the Web Environment: A Citation Analysis Using Google Scholar.‛ Grey Journal 4.2 (2008): 83-96. Koufogiannakis, Denise, Linda Slater, and Ellen Crumley. ‚A Content Analysis of Librarianship Research.‛ Journal of Information Science 30 (2004): 227-239. Ponti, M. ‚A LIS Collaboratory to Bridge the Research-Practice Gap.‛ Library Management 29.4/5 (2008): 265-277. Association of Research Libraries and Scholarly Publishing and Academic Evidence Based Library and Information Practice 2009, 4:2 142 Resources Coalition. Create Change 12 Feb. 2009. Monastersky, Richard. ‚The Number That’s Devouring Science.‛ The Chronicle of Higher Education 52.8 (2005): A12. Morrison, Frances P., and Li Li, Albert M. Lai, George Hripcsak. Repurposing the Clinical Record: Can an Existing Natural Language Processing System De- identify Clinical Notes? Journal of the American Medical Informatics Association 16.1 (2009): 37-39. Neuhaus, Christoph and Hans-Dieter Daniel. “Data Sources for Performing Citation Analysis. An Overview.‛ Journal of Documentation 64.2 (2008): 193-210. Rabina, Debbie L. ‚A Scholarly Communication Perspective of Grey Literature in Library and Information Science Education.‛ The Grey Journal 4.1 (2008): 34-40. Sales, Esther, Sara Lichtenwalter, and Antonio Fevola. "Secondary Analysis in Social Work Research Education: Past, Present, and Future Promise." Journal of Social Work Education 42.3 (Fall 2006): 543-558. Silva, Elizabeth B. ‚What’s *Yet+ To Be Seen? Re-Using Qualitative Data.‛ Sociological Research Online 12.3 (2007). 10 Feb. 2009 . Smith, Emma. "Pitfalls and Promises: The Use of Secondary Data Analysis in Educational Research." British Journal of Educational Studies 56.3 (Sep. 2008): 323-339. 18 Feb. 2009 Weiler, Derek. - "Libraries face "skills gap": 48 % of librarians could retire by 2005." Quill & Quire 66.1 (Jan. 2000): 10-11. Wilder, Stanley. Demographic Change in Academic Librarianship. Washington, DC: Association of Research Libraries, 2003.